collaborative time sensitive targeting aka technologies for team supervision march 31, 2008 missy...
TRANSCRIPT
Collaborative Time Sensitive Targeting
aka
Technologies for Team Supervision
March 31, 2008
Missy CummingsHumans & Automation Lab
MIT Aeronautics and Astronauticshttp://halab.mit.edu
Review of Original ProposalReview of Original Proposal
Project Goal Develop technological support for futuristic unmanned aerial vehicle (UAV) teams
engaged in TST operations (but really any time critical resource allocation task)
Technology to Support the Team Supervisor:
Activity Awareness Displays
Interruption Recovery Tools
Technology to Support UAV Operators:
Activity Awareness Displays
Objective Performance Measures
Teamwork in UAV OperationsTeamwork in UAV Operations
Intelligence Consumers (e.g., Ground Troops)
UAV Operators
Operations CenterGround Crew
Vehicle(s)
Generalizable to Other DomainsGeneralizable to Other Domains
3 “Confederates”as UAV Operators
Team Interface Issues in Time-Critical Team Interface Issues in Time-Critical OperationsOperations
• Increasing reliance on collaboration tools- e.g., email, instant messaging
(chat)- Increased communication
overhead
• Need interfaces that intelligently share activity information to improve planning & coordination in networked teams
- Activity awareness
• Research for technology aids to assist supervisors of teams virtually non-existent
• Measurement of technology interventions very difficult- Objective team performance
metrics are elusive
Boeing Team Test FacilityBoeing Team Test Facility
3 “Confederates”as UAV Operators
Simulated Task EnvironmentSimulated Task Environment
Situation Map Display Mission Status Display Remote Assistance Display
Surveillance support for a ground convoy through a hostile region
• UAV team consists of:
- 1 Mission Commander
- 3 UAV operators, each controlling multiple UAVs
• UAVs have camera sensors only
- Team must coordinate with external strike team to destroy identified threats
Team Supervision Decision AidingTeam Supervision Decision Aiding
UAV Operations Team Interruption recovery
tools for the team supervisor
Interruption Recovery Assistance (IRA) toolInterruption Recovery Assistance (IRA) tool
• Interactive event timeline, containing event “bookmarks”
• Increased recovery time but positive impact on decision accuracy, especially in complex task situations.
• IRA tool tended to provide greater benefits to participants without military experience
• Some experimental/interface issues so additional study needed
• Summer 2008 w/ Waterloo
Current Work: Predictive Tool for Team Current Work: Predictive Tool for Team SupervisionSupervision
UAV Team SupervisorTechnologies for
assessing:
- How well is the team doing?
- When & who should received attention?
Team
• The need for more objective team performance metrics
- Our focus is supervisory control in team settings with embedded layers of automation
• Predicting team states versus actual performance
- Artificial intelligence, behavioral pattern detection and performance correlations
• Alert team supervising agent of sub-optimal team behavior/cognitive strategies
- Relying on human knowledge-based reasoning to determine if an anomalous team state exists
• Two stage approach
- Predict an individual operator’s state transition as a proof of concept
- Adapt model to UAV team setting
Current Work: Team Performance PredictionCurrent Work: Team Performance Prediction
• Evaluation and predictions of team behaviors:- Automatic, continuous, and in real-time
• Only relies on easily observable data- User interaction with machine
- User communication
- Eye tracking?
• Use Bayesian pattern prediction methods applied to individual/team behaviors- Hidden Markov Model
• Not predicting performance per se- Anomalous conditions
- Value of human judgment
Team Prediction Model CharacteristicsTeam Prediction Model Characteristics
• Probabilistic state transitions
• Observables vs. Hidden States- Hidden states are not directly visible, but variables influenced by them are.
- Cognitive states vs. behaviors
• HMM uses the observables to infer:- Most likely hidden state sequence- Based on future sequences, the most likely observables- Machine learns the clustering of behaviors (need lots of data!)
Hidden Markov ModelsHidden Markov Models
Modify Flight Parameters
CommsCamera Control
Waypoint Update
H&S MonitoringTarget Monitoring
Mission Replan
Predicting Individual Behavior Predicting Individual Behavior
Human State Estimation - TRACS Human State Estimation - TRACS
Building the Model Building the Model
Group ofCriteria
IndividualCriterion
Group ofMatches
IndividualMatch
DataCluster
DataItem
Evaluate
BacktrackBrowse
SelectFilte
r
Automatch
Create
1 2 3 4 5 6
8 9 10 11 12 13
15 16 17 18 19 20
22 23 24 25 26 27
28 29 30 31 32 33 34
35 36 37 38 39 40 41
0
7
14
21
GrammarPattern
Recognizer/Predictor
Low Level Input
Future TeamBehavior
ILDs
4 State HMM for Mission Planning Task 4 State HMM for Mission Planning Task
• Unsupervised vs. supervised model training for state recognition
Group ofCriteria
IndividualCriterion
Group ofMatches
IndividualMatch
DataCluster
DataItem
Evaluate
BacktrackBrowse
SelectFilte
r
Automatch
Create
1 2 3 4 5 6
8 9 10 11 12 13
15 16 17 18 19 20
22 23 24 25 26 27
28 29 30 31 32 33 34
35 36 37 38 39 40 41
0
7
14
21
Leveraging Automation
ManualPlanning
Exploration
Predicting an Operator’s Strategies Predicting an Operator’s Strategies
Live Demo! Live Demo!
Validating our HMM Approach Validating our HMM Approach
• Currently only behaviors that entail mouse clicks are observed
- A limit, particularly for monitoring interfaces
• Eye tracking arguably gives us additional data about the information operators access
- Noisy
- Cost-benefit analysis?
• Study will be conducted this summer comparing HMM results with and without eye tracking input
Extending our Approach to UV Teams Extending our Approach to UV Teams
Visualization task
Evaluation/
Choice of UV + engage
Choice of UV
Sector A
Sector B
Sector C
a) Replan Path
b) Replan Target
Extending to task to teams: UV hand-off between Sectors
The Plan The Plan
• Develop HMM using experimental data from individual experiments
- Investigate supervised vs. unsupervised learning
• Develop a team version of the interface
• Update HMM with team experiments to be conducted this summer
Additional Future Work: Designing for Team Additional Future Work: Designing for Team Activity Awareness Activity Awareness
UAV Operations Team
Preliminary UAV Operator Display DesignsPreliminary UAV Operator Display Designs
Operator 1
UAV 4
Reroute/ReassignmentTarget ID Task
3UAV1
UAV 3 UAV 5
Health Status
Fuel level: 40%
Connected
- Searching - Searching Health Status
Fuel level: 70%
Health Status
Fuel level: 70%
Connected Connected 0:00:300:00:050:03:59 UAV3
5
Help
Request sent to:
UAV3
4 - Searching
Assistance Requests Overview
Req 3 (Complete)
Request from: Op 1 Elapsed time: 0m35sDetected by: UAV 3 Target type: 5 (Com Eq)
Request from: Op 3 Elapsed time: 0m30sDetected by: UAV 8 Target type: 5 (Com Eq)
Req 5 Request from: Op 1 Elapsed time: 0m05sDetected by: UAV 2 Target type: 3 (IED-V)
Req 4
ATR Information:
Possible Match: Target Type 5: Comm Equipment
Match Confidence: Medium
Found by UAV2 – Location: 43:00E/24:00N
ATR Decision:
SubmitClassification
Confirm ATR Suggested Classification:
Target Type 5 with Medium Confidence
Provide Alternate Target Classification:
Select Target Type…
Select Confidence Level…
Select Target Priority…
Veto Classification:
Image does not contain known target
Operator 3
Cancel
All
Others
Op 1
mCDR
Op 2
Classification:
Target Type 5 High Confidence
(no classification available)
Elapsed time:
0:00:20
Image Received at:
12:56:40
Request new image
Threat distance from the convoy: 15 miles
Elapsed Time:
Current Time:
1: 25: 30 PM
0: 22: 45
Mission Info:
0: 03: 28
UAV Reassignment
Elapsed Time:
Remaining Time:
0: 02: 37
Routes:
View Filter:
Hazards
Connection Blackouts
Targets Range
All
UAV 4
UAV 5
UAV 6
Sendprivate
All
Op 1
Op 3
mCDR
Strike team
others
Type here…
Op 2 – communication box
ALERTS & Clarification
SendType here…
Share Files
mCDR OthersStrike T Op 1 Op 3
Comm Status
0:03:57
On line
Handoff Available[0:00:30] Conference call
[0:01:20] mCDR to OP1:
[0:01:22] Op1 to Op3:
[0:01:25] Op3 to strike team:
[0:01:29] Op3 to Op2:
4 - Searching
Repair
Repair
RepairRepair
Health Status
Health Status
Health Status
Fuel level: 40%
Fuel level: 70%
Fuel level: 70%
- Target ID5
6
Connected
Connected
DisconnectedSafety route in 0:29:59
Reconnecting in 0:00:49
0:00:18
0:00:04
0:03:57
(0:01:50) UAV 7 (area3) has been shot down
(0:03:00) UAV 4 Fuel Level at 40%
(0:03:40) UAV 6 enter in a difficult area to connect
mCDR
Strike team
OP1
Others
OP3
0:03:57
0:03:57
0:01:53 - Changing shift
0:00:33 - Target task
Comm
Comm
Comm
Comm
0:03:57
Repair
Repair
Repair
Repair
Repair
Comm
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
Very GoodSpeed: X Mb/sMIT Network
MapDisplay
CommunicationsDisplay
Tasking Display:TargetID & rerouting, reassigning UAVs
DeliverablesDeliverables
• 6 Conference papers- ASNE, HSIS, ICCRTS, HFES
• 4+ Technical reports• 3 Undergraduate theses
- 2 pending• 1 Masters thesis• 1 PhD dissertation (pending)• 3 Journal articles in various stages• 1 workshop (CSCW 2006)• IP disclosure pending HMM validation• The Holy Grail of team research???
• Near Term
- Eye tracking study
- HMM-Multi UV experiment
- Extension for supervisor IRA tool
• Mid Term
- Activity awareness displays for operators
- Decision support display for HMM
- Multi-UV connection to predictive model system architecture research (ONR)
Development of CE metric
• Project slated to end December 2008- Requested no-cost extension to JUN 09
- Additional funding
Future PlansFuture Plans
Backup Backup
Simulated Task Environment Display DetailSimulated Task Environment Display Detail
Map Display
Mission Status Display
Simulated Task Software ArchitectureSimulated Task Software Architecture
Large-Screen Wall
Displays
Situation Map Display Mission Status Display
TabletPC Display
Mission Commander Display
Simulation & Collaboration Server
(Grouplab SharedDictionary)
Situation Map Display
Mission Status Display
Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy safety
• Current & expected operator task performance, relative to convoy safety
4
(nominal)
(down)
8
1
(reviewing ATR imagery)
Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness
Potential threat
envelopeTarget strike indicators
Known threat
envelope
Situation Map Display
Mission Status Display
• UAV status & tasking
• Current & expected convoy safety
• Current & expected operator task performance, relative to convoy safety
Situation Map Display
Mission Status Display
Designing to Promote Activity AwarenessDesigning to Promote Activity Awareness
• UAV status & tasking
• Current & expected convoy safety
• Current & expected operator task performance, relative to convoy safety
Developing Methodology for Deriving Developing Methodology for Deriving Collaborative System Requirements Collaborative System Requirements
Project Goal:
• To develop techniques to identify dependencies in operator decision making to understand how to assist coordination of team member tasking