don strausberger dr. stephen balakirsky georgia tech ... tiw/proceedings/strausberger.… ·...
TRANSCRIPT
Distributed T&E of Autonomous Systems:
Challenges and Opportunities for LVC Testing
Georgia Tech Research Institute
May 2016
Don Strausberger
Dr. Stephen Balakirsky
2
Ongoing Autonomy Science & Technology Programs (1 of 3)
Loyal Wingman
Domains Ongoing S&T Efforts
Air
Description and Reference
For both demonstrations, the unmanned aircraft is expected to safely operate in a cooperative/team configuration with a 5th generation manned aircraft from a fixed airfield, accomplish mission objectives, and return …shall be able to operate untethered from the ground without full-time direction from the manned aircraft. RFI-AFRL-RQKH-2015-003.pdf
Using collaborative autonomy, CODE-enabled unmanned aircraft will find targets and engage them as appropriate under established rules of engagement, leverage nearby CODE-equipped systems with minimal supervision, and adapt to dynamic situations such as attrition of friendly forces or the emergence of unanticipated threats.http://www.darpa.mil/program/collaborative-operations-in-denied-environment
3
Ongoing Autonomy Science & Technology Programs (2 of 3)Domains Ongoing S&T Efforts
CARACAS
Sea
Description and Reference
http://www.darpa.mil/program/anti-submarine-warfare-continuous-trail-unmanned-vessel
http://www.onr.navy.mil/Media-Center/Press-Releases/2014/autonomous-swarm-boat-unmanned-caracas.aspx
… allows unmanned Navy vessels to overwhelm an adversary. Its sensors and software enable swarming capability, giving naval warfighters a decisive edge.
Advance unmanned maritime system autonomy to enable independently deploying systems capable of missions spanning thousands of kilometers of range and months of endurance under a sparse remote supervisory control model. This includes autonomous compliance with maritime laws and conventions for safe navigation, autonomous system management for operational reliability, and autonomous interactions with an intelligent adversary.
4
Ongoing Autonomy Science & Technology Programs (3 of 3)
Domains Ongoing S&T Efforts
xUUVUn
de
rse
a
Description and Reference
http://www.navaldrones.com/LDUUV-INP.html
The LDUUV prototype's navigation algorithms and sense-and-avoid technologies are planned ….
…develop a large unmanned submersible able to conduct missions longer than 70 days in open ocean and littoral seas..
…missions will include ISR, acoustic surveillance, Anti-submarine Warfare, mine counter-measures, and offensive operations…
5
The T&E Challenge
How am I as a Test Professional going to generate sufficient evidence through Test & Evaluation to assure the Commander of the following:
His/her newly deployed fleet of 40 fully autonomous surface vessels (actively engaged in 90 day continuous anti-submarine warfare missions) will not cause damage to their own fleet or be the center of an international incident as a result of at sea collisions.
His/her newly deployed Loyal Wingman (an autonomous 5th Generation fighter) will maintain air collision avoidance protocol, execute evasive maneuvers as necessary, and establish the requisite trust in the lead pilot.
His/her newly deployed UUV’s will consistently and effectively recognize threatening situations and evade intelligent adversaries without any human intervention.
Wingman
Sub Hunter
Mine Hunter
Examples
6
Autonomy Testing – Tackling the Elephant coming to the range near you!
Purpose• Drive thought/discussion in the Test Community
towards innovative Live Virtual Constructive (LVC) solutions to solve autonomy-unique test challenges
Agenda• Definitions & Terminology
• When/why is autonomy testing different?• Human Aspects/Assurance Arguments
• Relevant LVC Test Environment Decomposition• Live Infrastructure• Virtualization Infrastructure• Test Control Infrastructure
• 1st Order Analysis (Bandwidth Considerations)
• Conclusion
7
Autonomy Definition*
Automation
The system functions with no/little human operator involvement; however, the system performance is limited to the specific actions it has been designed to do. Typically these are well-defined tasks that have predetermined responses.
Rule-based responses
Autonomy
The system has a set of intelligence-based capabilities that allow it to respond to situations that were not pre-programmed or anticipated prior to deployment. Autonomous systems have a degree of self-government and self-directed behavior.
Decision-based responses
Our focus for this presentation is on the T&E of systems that perform autonomous route planning and must support collision avoidance and evasion of hostile entities.
*As defined in the DoD Autonomy Community of Interest (COI) Test and Evaluation Verification and ValidationWorking Group Technology Investment Strategy 2015-2018
8
Terminology
Situational AwarenessSituational UnderstandingSituational Assessment
PerceptionComprehensionProjection
ReflectiveDeliberativeReactive
Goal-basedRule Based
BeliefsDesiresIntent
Different disciplines have developed different terminology in closely related areas
Robotics HumanFactors
AI
Others…
Terminology does not map one-to-one across disciplines, however they can be associated with the OODA loop to provide a common reference.
9
Why is Autonomy Testing Different?
> Human Values > Trust> Empathy > Compassion> Fight or Flight > Cost and Risk … …
Several of the core conditions that guide a human’s decision making on a daily basis are unaccounted for in today’s autonomy. This fundamental aspect of autonomy significantly impacts the “acceptance criteria” of autonomy, and indirectly (but significantly) impacts T&E.
> Memory > Multi Sensor Interpretation
> Projection > Judgement> Sensing > Skilled Movements… …
Human Machine
10
The “Assurance Argument”
FACT: There is an additional burden of proof required for the acceptance of autonomy that controls platforms that can drive/fly/propel themselves into others, that were previously controlled by humans, and must operate in less constrained environments.
• Collision avoidance and hostile evasion will be common requirements of the autonomous route planning for all these systems.
• Generating sufficient “evidence” that these platforms WILL perform appropriately in operational environments places significant demands on the Test Methodology and Infrastructure to comprehensively evaluate collision avoidance and hostile evasion.
PREMISE: Live Virtual Constructive test environments with the ability to inject virtual entities into the live SUT’s perception engine will be required to generate sufficient credible evidence before these systems will be accepted.
11
Basic View of Surface LVC Test Environment
Geography Environment
SUT Position, Speed, Heading
SUT IntentLive NPC’s
ExpectedSUT Intent
Expected SUT Position, Speed, Heading
Scenario
Model or Specification
Resultant Behavior
Desired Behavior
Compare and Deceide
Test Analysis
Live SUT
Virtual NPC’s
NPC -Non Player Character: A dynamic entity that may influence, but is not under the control of the autonomy.
12
Live Environment
System Under Test
Introspection Data
SUTAutonomy
Live Sensors
Pose
Actual Vessel
Behavior
Live Sensors
Actual Vessel
Captain’s behavior
Pose
Live NPC’s
Pose: A objects position and orientation relative to some coordinate system.
13
Virtualization Infrastructure
Virtualization Infrastructure
Pose
Sensor Model
Pose
Sensor Model
BehavioralModel
BehavioralModel
VesselModel
VesselModel
Coord.Transform
SUT Sensor Model
Coord.Transform
vNPC 1 vNPC 2
User Input
Scenario Generation
Test Analysis
Compare and Deceide
Virtual NPC Parameters
Geography & Environment
Test Control Infrastructure
Model/Specification
Desired Behavior
Resultant Behavior
14
Resulting LVC Infrastructure
System Under TestVirtualization Infrastructure
User Input
Scenario Generation
Test Analysis
Compare and Deceide
Virtual NPC Parameters
Geography & Environment
Test Control Infrastructure
Model/Specification
Pose
Sensor Model
Pose
Sensor Model
BehavioralModel
BehavioralModel
VesselModel
VesselModel
Live Sensors
Actual Vessel
Captain’s behavior
POSE
Introspection Data
SUTAutonomy
Live Sensors
Pose
Coord.Transform
Live NPC’s
Desired Behavior
Resultant Behavior
Live Environment
SUT Sensor Model
Coord.Transform
vNPC 1 vNPC 2
Actual Vessel
Behavior
Comms/Spectrum/Data(TENA)Infrastructure
15
Additional Autonomy-Driven Scenarios To Consider
Scenario to Consider Test Example Description
Could LVC be used to cost-effectively scale and execute live testing of 1 SUT versus M non-player characters?
Testing a UAV Swarm’s collision avoidance in A2D2 environments, complex USV ColRegs, etc.
1-vs-M Testing
Could LVC be used to cost-effectively scale and execute live testing N SUTs versus 1 non-player character?
Testing collaborative autonomous blue-force USV interdiction of red USV sea –based penetrator or UAV interception of red UAV airbornepenetrator
N-vs-1 Testing
Could LVC be used to cost-effectively scale and execute live testing N SUTs versus M non-player characters?
Complex swarm-vs-swarm live/virtual mix of blue SUTs versus red NPCs (some live some virtual) with various penetrator and interdiction behavior assignments.
N-vs-M Testing
What are the bandwidth demands (to/from the live SUT) required to support these scenarios with LVC?Are these bandwidth demands within the ballpark of practicality, or do they drive us toward a radically different approach?
• SUT Speed/Agility - faster/more agile SUT platforms require higher sample rates (position estimate error)
• Number of NPCs – testing performance against “swarming” red forces increases data rates
• Number of SUTs – testing collaborative autonomous blue force UAVs/USVs
Bandwidth Drivers
16
17
0
100
200
300
400
500
600
700
800
900
1000
2 4 8 16
BA
ND
WID
TH (
KB
PS)
NUMBER OF NPC'S
1-vs-2 to 1-vs-16 Bandwidth Estimate
25.00% 50.00% 75.00% 100.00%
1 vs M Engagement Scenario
Consider an air-to-air collision avoidance/hostile evasion test scenario between a UAV (SUT) versus 2 to 16 NPCs
Blue UAV avoidance in A2D2 / red UAV aggressor scenario’s
SUT/NPCs @ 300 mph
< 1 Mbps!
18
N vs 1 Engagement Scenario
Consider a sea surface based collaborative blue force USVs performing an intercept test scenario
Blue USV interdiction of a hostile boat, UAV team interception of red airborne penetrator
SUTs/NPC @ 40 mph (~ 35 knots) – USV example
< 150 Kbps!
0
20
40
60
80
100
120
140
2 4 8 16
BA
ND
WID
TH (
KB
PS)
NUMBER OF SUT'S
2-vs-1 to 16-vs-1 Bandwidth Estimates
25.00% 50.00% 75.00% 100.00%
19
N vs M Engagement Scenario
Consider a UAV “Swarm vs Swarm” test scenario
SUTs/NPCs @ 40 mph (~ 35 knots)
< 2 Mbps!
0
200
400
600
800
1000
1200
1400
1600
1800
4 20 50 100
BA
ND
WID
TH (
KB
PS)
NUMBER OF NPC'S + SUT'S
2-vs-2 to 50-vs-50 UAV Bandwidth Estimates
25.00% 50.00% 75.00% 100.00%
• First-order “back of the napkin” calculations show bandwidth demands between the Live SUT and the Virtualization Infrastructure are well within the realm of the ‘possible’
• e.g. could be readily supported by commercially available mobile ad hoc radio network communication systems
• Additional Considerations/Further investigation
• Bandwidth
• Network Efficiency
• Protocol Efficiencies (TENA, DDS, …)
• Ad Hoc Network Efficiencies (forwarding, latencies, …)
• Robustness/redundancy (message dropouts)
• SWaP on small platforms (RF link budgets, amplifiers, etc.)
• Information Assurance & Encryption
• Value of Standardization
• Introspection Data
• Pose exchange
• Virtualization Infrastructure
Technical Summary
20
21
The Power of LVC Constructs in live testing of autonomous route planning
Test Challenge Value Add Key Domains
AssuranceArgument
LVC may enable execution of tests with live SUTs that would otherwise (due to safety) only be conducted in simulation, generating increasingly credible evidence of proper collision avoidance and hostile evasion behaviors for programs that include autonomous route planning
Air, Sea Surface, Ground,
Undersea
1-vs-M Testing LVC constructs may be required to cost-effectivelyscale and execute live testing of 1 SUT versus M non-player characters(NPCs = platforms the SUT must avoid or evade)
Air, Sea Surface
N-vs-1 Testing LVC constructs may be required to cost-effectivelyscale and execute live testing of N SUTs versus 1 non-player character (multiple autonomous collaborative SUT’s serving to interdict a threat (the NPC))
Air, Sea Surface
N-vs-M Testing LVC constructs may be required to cost-effectivelyscale and execute live testing of N SUTs versus M non-player characters(NPC(s) = platform the SUT(s) must interdict)
Air, Sea Surface
