Christopher Bove (RBE)

Alexander Wald (CS)

Advised by:

William Michalson (WPI, ECE)

Mark Donahue (MITLL, 76)

Jason LaPenta (MITLL, 76)

Collaborative Robotics Heads-up Display Major Qualifying Project

This material is based upon work supported by the Department of the Navy under Air Force Contract No. FA8721-05-C-0002 and/or FA8702-15-D-0001. Any opinions,

findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the Department of the Navy.

DISTRIBUTION STATEMENT A. Approved for public release: distribution unlimited.

C. Bove, A. Wald 10/12/2016

Slide 2

Research Goals

• Support multiple platforms

• Robust, real-time data processing

• Intuitive interfaces for data interaction

• Intelligent autonomous behavior

Problem: Harness Talents of Humans and Autonomous Systems

• Dirty, Dull, and Dangerous Jobs

• Hazard Agnostic

• Enhanced Sensing - Produce LOTS of data!

• Remote Operations

• Expendable and Replaceable

Autonomous Systems

Humans

• Excellent High Level Decision Making

• Fast Task Distribution and Planning

• Ethical Decisions

• Adaptive in Uncertainty

Merge

via Hybrid

Human Robot

Teaming

Distill Information and Present Intuitively to User

C. Bove, A. Wald 10/12/2016

Slide 3

Relies on pose estimate to relate user with environment.

Augment Reality on Heads-Up Displays to Improve Collaboration and Situational Awareness

Display Augmented Reality Objects

to user through HUD.

C. Bove, A. Wald 10/12/2016

Slide 4

• Inertial Navigation System (INS) – Inertial Measurement Unit (IMU) and GPS

– IMU provides orientation: magnetometer, accelerometer, and gyroscope

– GPS for position

– Position unavailable with obstructed view of sky

• Simultaneous Localization and Mapping (SLAM) – Determine current position by examining world

– Use further examinations to build map

– Close the loop by recognizing previously visited areas

• Hector SLAM utilizing LIDAR scans – Measure ranges and particle filter for probable

locations, provides X,Y, Yaw

– Can utilize IMU or spinning LIDAR to develop 3D pose and mapping

– LIDAR has large SWaP, emits energy into environment

Variety of Methods for Obtaining Pose

Hector SLAM Utilizing LIDAR Sensor

C. Bove, A. Wald 10/12/2016

Slide 5

Monocular Visual Simultaneous Localization and Mapping

Oriented FAST and Rotated Brief (ORB) SLAM – Feature based mapping, sparse mapping – Robust pose determination and scale-drift

aware loop closure – Focus of work for MQP

Large Scale Direct (LSD) SLAM – Gradient pixel based stereo depth-matching – Dense point cloud generation, processor

intensive

C. Bove, A. Wald 10/12/2016

Slide 6

Problem: Maps with Monocular Vision Lack a Real Scale

Frame at t2

C. Bove, A. Wald 10/12/2016

Slide 7

Problem: Maps with Monocular Vision Lack a Real Scale

Possible

Distances

An unknown baseline between two

images results in an unknown

distance to objects.

Frame at t1 Frame at t2

Possible Baselines

Feature

C. Bove, A. Wald 10/12/2016

Slide 8

Envisioning the Future of Augmented Reality

Research Grade Equivalents to ODG Glasses + LIDAR for Truth

Objective System

Research System

Tasks for MQP:

Data Collection

Scale and Scale Drift Correction

Runtime Performance Evaluation

C. Bove, A. Wald 10/12/2016

Slide 9

Goal: Determine Position with Prototype Helmet Mapping System

Orthogonal

Color Cameras

Inertial Navigation

System

ODG HUD

Glasses

NVIDIA Jetson TX1

Embedded 15W

System On Chip

Portable Data Collection

tethered to Helmet

LIDAR (Truth Data) GPS Receiver

C. Bove, A. Wald 10/12/2016

Slide 10

The Big Picture Approach

Our Project

Dense 3D Point Cloud

C. Bove, A. Wald 10/12/2016

Slide 11

Fuse IMU Data into ORB SLAM to Correct Scale and Drift

Our Project

C. Bove, A. Wald 10/12/2016

Slide 12

Envisioning the Future of Augmented Reality

Research Grade Equivalents to ODG Glasses + LIDAR for Truth

Objective System

Research System

Tasks for MQP:

Data Collection

Scale and Scale Drift Correction

Runtime Performance Evaluation

C. Bove, A. Wald 10/12/2016

Slide 13

Collected Data to Check Assumptions and Evaluate Performance

• Tests of several motion patterns

– Static

– Linear

– Circle

– Square

– Erratic

– Head Turns

– Jumping

• Rotational Testing

– Increasingly fast rotations

• Sanity checks for coordinate systems

– XYZ translation

– Roll, pitch, yaw tests

– 360 degree RPY tests

C. Bove, A. Wald 10/12/2016

Slide 14

Collected Data to Check Assumptions and Evaluate Performance

• Tests of several motion patterns

– Static

– Linear

– Circle

– Square

– Erratic

– Head Turns

– Jumping

• Rotational Testing

– Increasingly fast rotations

• Sanity checks for coordinate systems

– XYZ translation

– Roll, pitch, yaw tests

– 360 degree RPY tests

C. Bove, A. Wald 10/12/2016

Slide 15

• Sources of Drift:

– Gravity

– DC biases

• Approach:

– Calculate gravity vector from orientation

– Calculate DC bias using a calibration phase

Implemented Gravity-Compensated Dead Reckoning on IMU

Accelerometer

DC Bias

Gravity Vector

C. Bove, A. Wald 10/12/2016

Slide 16

Envisioning the Future of Augmented Reality

Research Grade Equivalents to ODG Glasses + LIDAR for Truth

Objective System

Research System

Tasks for MQP:

Data Collection

Scale and Scale Drift Correction

Runtime Performance Evaluation

C. Bove, A. Wald 10/12/2016

Slide 17

Determined Transformations Between Systems

C. Bove, A. Wald 10/12/2016

Slide 18

Scaled Initial Baseline Estimate to Fix Scaling

Transform

Mocap or IMU

Position

ORB SLAM

Frames

Transform

Inject Extra

Pose Data into

Frames

Transform

Mocap or IMU

Position

ORB SLAM

Frames

Transform

Scale

Transformation

to Match

Scale Map Points

with Same Scaling

C. Bove, A. Wald 10/12/2016

Slide 19

• Modified ORB SLAM implementation

• Deliver inertial velocity and absolute orientation

Corrected Motion Model and Injected Absolute Orientation to Mitigate Drift

Mocap or IMU

Position

Last Pose x Velocity Mocap or IMU

Orientation

C. Bove, A. Wald 10/12/2016

Slide 20

Results

C. Bove, A. Wald 10/12/2016

Slide 21

IMU Gravity Compensated and Position Integrated

C. Bove, A. Wald 10/12/2016

Slide 22

Results: Coordinate Systems Correctly Aligned

Transform

Heading Error:

50.2 degrees

Correct Heading to Object

C. Bove, A. Wald 10/12/2016

Slide 23

ORB Scale Correction Achieved Accurate Results: ~10cm Error w/ Truth Data Injected

Heading Error:

2.66 degrees

Correct Heading to Object

C. Bove, A. Wald 10/12/2016

Slide 24

Scale Drift Correction Produced Mixed Results

Before Drift Correction After Drift Correction

Implemented Method to Correct Drift. Future Refinements are Necessary.

C. Bove, A. Wald 10/12/2016

Slide 25

Envisioning the Future of Augmented Reality

Research Grade Equivalents to ODG Glasses + LIDAR for Truth

Objective System

Research System

Tasks for MQP:

Data Collection

Scale and Scale Drift Correction

Runtime Performance Evaluation

C. Bove, A. Wald 10/12/2016

Slide 26

• Real time – System eventually on Jetson TX1

• Analyzed ORB SLAM run time performance

• Step 1: Compare each thread’s average cycle duration

Conducted System Performance Evaluation of ORB SLAM

Frame Received

Tracking Thread

Local Mapping

Loop Closing

Key Frame Key Frame

Loop Closing

Key

Frame Processing

Local Bundle Adjustment

Global Bundle Adjustment

C. Bove, A. Wald 10/12/2016

Slide 27

• Step 2: Analyze specific tasks in call hierarchy

Conducted System Performance Evaluation of ORB SLAM

Tracking Thread

Track

Monocular

Grab Image

Monocular

Cvt Color Track

Track Reference

KeyFrame

Track Local

Map

Track With Motion

Model

a) Find average duration for each task

b) Find total child time for each node

Frame

Extract ORB

Construct

Frame

C. Bove, A. Wald 10/12/2016

Slide 28

ORB Performance Evaluated and Bottlenecks Exposed

Track

Monocular

Grab Image

Monocular

Frame

Extract ORB

Construct

Frame

195 sec

230.5 sec

195 𝑠𝑒𝑐

230.5 𝑠𝑒𝑐= 0.8460 = 85% of run time

230.5 sec (total)

195 sec

𝟖𝟓% 𝐨𝐟 𝐓𝐨𝐭𝐚𝐥 𝐑𝐮𝐧 𝐓𝐢𝐦𝐞

…

…

C. Bove, A. Wald 10/12/2016

Slide 29

• Scale correction

– With truth, worked well

– With inertial position, dead reckoning drift impacted accuracy

– Staking the scale at one point does not introduce feedback

• Scale drift correction

– Created internal inconsistencies between perceived motion in map and estimated pose

– Produced jagged motion as estimates switched

– Need to optimize position of points or mix estimates externally

Conclusion: Passive Sensor Position Estimate Achievable

• Performance evaluation

– 84.60% of tracking run time spent on ORB feature extraction

– GPU optimized calls exist

C. Bove, A. Wald 10/12/2016

Slide 30

Future Improvements to Provide Robustness

Frame Filter

C. Bove, A. Wald 10/12/2016

Slide 31

Thank you!

C. Bove, A. Wald 10/12/2016

Slide 32

• https://www.ptgrey.com/flea3-20-mp-color-usb3-vision-e2v-ev76c5706f-3

• https://developer.nvidia.com/sites/default/files/akamai/embedded/images/jetsontx1/JetsonTX1_DevKit_3Quarter_ON.png

• http://www.govconnection.com/shop/samsung-512gb-ssd-950-pro-pcie-3.0-nvme-m.2-2280-internal-solid-state-drive/30785312?cac=CartItem

• http://www.hokuyo-aut.jp/02sensor/07scanner/download/products/utm-30lx-ew/

• https://www.hokuyo-aut.jp/02sensor/07scanner/img/utm_30lx_top.jpg

• https://www.microstrain.com/sites/default/files/styles/larger__550x550_/public/gx4-45_0.jpg?itok=R4F6eE8L

• https://www.inition.co.uk/wp-content/uploads/2014/09/s250e1.png

• http://www.digitalartsonline.co.uk/cmsdata/reviews/3444105/MaxPNG688.jpg

Photograph Links

C. Bove, A. Wald 10/12/2016

Slide 33

Monocular Vision Accumulates Errors without Loop Closure

Scale-Drift Aware Loop Closure

And Optimization Corrects Path

C. Bove, A. Wald 10/12/2016

Slide 34

Sensors Capable of Providing Positions

Passive Sensors are Ideal for Use

in the Field and Mobile Systems

• LIDAR Systems:

– Directly provide accurate position data

– High cost

– Emit laser infrared energy

• Inertial Measurement Units:

– No energy emission

– Low power

– Low weight

– Only provide linear acceleration and angular velocity

• Global Position Systems:

– Accurate positioning on Earth

– Passive, could be interrupted

• Vision Systems

– No energy emission

– Low power

– Low weight

– Requires extensive computer processing