h ow m ulti - robot f oraging s cales with n umber of r obots prasanna velagapudi paul scerri, katia...
TRANSCRIPT
Speaking Qualifier - Nov. 24, 2009
HOW MULTI-ROBOT FORAGING SCALES WITH NUMBER OF
ROBOTS
Prasanna VelagapudiPaul Scerri, Katia Sycara
Robotics Institute, Carnegie Mellon University
Huadong Wang, Michael Lewis
Dept. of Information Sciences, University of Pittsburgh
Speaking Qualifier - Nov. 24, 2009 2
Outline
• Motivation• Problem• Related Work• Experiment• Results• Discussion/Conclusion
Speaking Qualifier - Nov. 24, 2009 3
Motivation
Perimeter PatrolsDisaster response Search and Rescue
Speaking Qualifier - Nov. 24, 2009 4
Motivation
• Goal: Increase operator’s span-of-control– Span-of-control: the number of subordinates (robots) a
supervisor (operator) has
• Human operators are necessary– Complex perception, meta-knowledge– Supply high-level, dynamic goals
• Humans get overloaded
Speaking Qualifier - Nov. 24, 2009 5
Problem: Improving Span-of-Control
• How can we increase human operators’ span-of-control?
– How does task performance vary with number of robots?
– Which tasks are most limiting to the operator?
– Does alleviating a task improve performance?
Speaking Qualifier - Nov. 24, 2009 7
Decomposing Foraging
Exploration Perceptual Search
Foraging is composed of two largely independent but concurrent subtasks:
Speaking Qualifier - Nov. 24, 2009 8
Focusing the Problem
• Multi-robot foraging with waypoint control– Widely cited as likely field application– Each robot searches its own region
• Minimal coordination to avoid overlaps or gaps– Waypoint control:
• Lowest level of automation compatible with independent control of multiple robots
Speaking Qualifier - Nov. 24, 2009 9
Related Work
Study Task World Robots InteractionTrouvain & Wolf (2002):User study of the impact ofrobot group size
Navigation 2D simulatedoffice world
2, 4, 8 UGVs(homogeneous)
Waypoint
Trouvain et al. (2003): User study of map based and camera based user interface
Exploration 3D simulatedoutdoor world
1, 2, 4 UGVs(homogeneous)
Supervisory + waypoint control
Olsen & Wood (2004): Fan-out independent study
Foraging 2D simulatedoffice like world
18 UGVs (homogeneous)
Waypoint
Humphrey et al. (2007):User study of robot team & halo interface
Foraging 3D simulatedoutdoor world(USARSim)
6, 9 UGVs(heterogeneous)
Teleoperation and scripted behaviors
Nehme et al. (2008): Impact of Heterogeneity on Operator Performance
Foraging 2D simulated 6, 8,12 UGVs(heterogeneous)
WaypointAutomation
Speaking Qualifier - Nov. 24, 2009 10
Related Work: Span-of-control Limits
• For foraging, operator span-of-control limits fall between 4-9 robots – Olsen & Wood (2004), Humphrey et al. (2007)
• Dependent on environmental demands• Generally, operators can use more robots if robots
have a higher level of autonomy
Speaking Qualifier - Nov. 24, 2009 11
Related Work: The Fan-out Plateau
If a robot is added to a team, the change in performance is proportional to robot’s independence and operator’s available cognitive resources.
– Olsen & Wood (2004)Pe
rfor
man
ce
# of Robots
Performance plateau when operator saturates
Diminishing returns as # of robots increases
Speaking Qualifier - Nov. 24, 2009 12
Fan-out Hypothesis
• Task performance should follow Fan-out model
Perf
orm
ance
# of Robots
Performance Plateau
Diminishing returns
Speaking Qualifier - Nov. 24, 2009 13
Subtask Hypothesis
• Task performance will reflect which subtask is contributing more to operator workload
Perf
orm
ance
# of Robots
Subtask 1
Subtask 2 (limiting)
Full task
Exploration
Perceptual Search
Speaking Qualifier - Nov. 24, 2009 14
Experiment
• Task: Simulated search and rescue• Single operator controlling 4, 8, and 12 robots
– Waypoint control (primary)– Direct teleoperation
• Perform either full foraging task or just one subtask (exploration or perceptual search)
Speaking Qualifier - Nov. 24, 2009 15
USARSim
[http://www.sourceforge.net/projects/usarsim]
• NIST-maintained open source simulator
• High-fidelity physics• “Realistic” rendering
– Camera– Laser scanner (LIDAR)– IMU/Odometry
Speaking Qualifier - Nov. 24, 2009 18
MrCSMulti-robot Control System
Waypoint Navigation
Teleoperation
Video/ Image Viewer
Status Window
Map Overview
Speaking Qualifier - Nov. 24, 2009 20
Conditions
Exploration• Issue waypoints
Perceptual Search• Locate victims
Full Task• Issue waypoints• Locate victims
Speaking Qualifier - Nov. 24, 2009 21
Experimental Procedure
• Between groups, repeated measure design
Full Task Exploration Perceptual Search
DemographicsTraining 20min4 robots 15min
Workload Survey
8 robots 15min Workload Survey
12 robots 15minWorkload Survey
Each
Sub
ject
Speaking Qualifier - Nov. 24, 2009 22
Methodology
• Independent Variables:– Conditions of Task– Numbers of Robots
• Dependent Variables:– Workload (NASA-TLX)– Victims found– Area Explored– Switches in focus among
robots– Number of assigned
missions– Average path length– Robots neglected
Speaking Qualifier - Nov. 24, 2009 23
Participants
• 44 paid participants from U. of Pitt. community– 25 male, 20 female– Ages 20-33 (average age 25.18)
• No prior experience with robot control• Most were frequent computer users
Speaking Qualifier - Nov. 24, 2009 24
Conditions
Exploration• Issue waypoints
Perceptual Search• Locate victims
Full Task• Issue waypoints• Locate victims
Speaking Qualifier - Nov. 24, 2009 25
Task Performance: Full Task vs. Perceptual Search
• Compare number of victims found in full task and perceptual search conditions
• Expected: Fan-out plateau and performance gapPe
rfor
man
ce
# of Robots
Perceptual search
Full task
Speaking Qualifier - Nov. 24, 2009 26
Task Performance: Full Task vs. Perceptual Search
F1,28 = 27.4
p < .0001
Diminishing return/plateau
Operator overload
Speaking Qualifier - Nov. 24, 2009 27
Task Performance:Full Task vs. Exploration
• Compare total area explored in full task and exploration condition
• Expected: Fan-out plateau and performance gapPe
rfor
man
ce
# of Robots
Exploration
Full task
Speaking Qualifier - Nov. 24, 2009 28
Task Performance:Full Task vs. Exploration
F1,28 = 21.17
p < .002
Diminishing return/plateau
Operator overload
Similar performance
Speaking Qualifier - Nov. 24, 2009 29
Fewer pathsLonger paths
4 Robots 8 Robots 12 Robots20
25
30
35
40
45
fulltask
Exploration Control Statistics
# Paths Issued Average Path Length
# of
Pat
hs
Leng
th o
f Pat
h (m
)
4
4.5
5
5.5
6
6.5
7
7.5
8
8.5
fulltask
4 robots 8 robots 12 robots
Speaking Qualifier - Nov. 24, 2009 30
4 Robots 8 Robots 12 Robots0
0.20.40.60.8
11.21.41.61.8
2
Neglected fulltaskNeglected exploration
Neglected Robots
• Neglected robots are never issued waypoints• Further evidence of reaching operator span-of-control limits
χ22 = 10.75
p = 0.005
# N
egle
cted
Rob
ots
Speaking Qualifier - Nov. 24, 2009 31
Number of Switches Between Robots
• Before operators issue waypoints or mark victims, they must switch to the desired robot
• Switches are cognitively expensive to operatorsExploration Perceptual Search
Speaking Qualifier - Nov. 24, 2009 32
Close correspondence
Number of Switches Between Robots
F2,54 = 12.6 p < .0001
Num
ber o
f Sw
itche
s Be
twee
n Ro
bots
Significant gap
10
15
20
25
30
35
40
45
50
55
perceptual search explorationfulltask
4 robots 8 robots 12 robots
Speaking Qualifier - Nov. 24, 2009 33
Operator Pause Statistics
• Robots become paused when commanded by operator or when finished with all waypoints– Perception pauses
• Operator stops robot in middle of mission to locate victim– Navigation pauses:
• Robot is paused by itself or by operator so that a new set of waypoints can be issued
• Pauses may indicate operator neglect
Speaking Qualifier - Nov. 24, 2009 34
Distribution of Pauses
Full Task Exploration Perceptual Search
4 8 12 44 8 812 12
# of Robots
Pause (sec)
More frequent pauses
More long-duration pauses
(robot neglect)
Coun
ts
Speaking Qualifier - Nov. 24, 2009 35
Aggregate Pause DurationAg
greg
ate
Paus
e D
urati
on p
er R
obot
(s
ec)
Speaking Qualifier - Nov. 24, 2009 36
Workload Survey
• Survey (NASA-TLX) administered after each session (4, 8, 12 robots)
• Aggregate score indicates participants’ subjective assessment of cognitive workload
• Expected:
Wor
kloa
d# of Robots
Subtask 1
Subtask 2 (limiting)
Full task
Speaking Qualifier - Nov. 24, 2009 38
Discussion
Fan-out Hypothesis: Task performance should follow Fan-out model
• Performance in subtasks consistent with fan-out
• Performance in full task does not follow fan-out– Inflection point in performance at 8 robots– Operators experiencing cognitive overload
Speaking Qualifier - Nov. 24, 2009 39
Discussion
Subtask Hypothesis:Task performance will reflect which subtask is contributing more to operator workload
• Exploration and full task performance similar– Removing perceptual search subtask removes little
workload• Perceptual search performs better than full task
– Significantly less workload than other tasks with 12 robots
Speaking Qualifier - Nov. 24, 2009 40
Conclusions
– How does task performance vary with number of robots?Fulltask: operators overload, performance dropsSubtasks: follows fan-out model
– Which tasks are most limiting to the operator?The exploration subtask
– Does alleviating a task improve performance?Removing exploration helps, removing perceptual search does not
Speaking Qualifier - Nov. 24, 2009 41
Conclusions
• How can we increase human operators’ span-of-control?
• Offload (automate) Exploration subtask– When operators are overloaded with full task, performance
drops dramatically– Operators devote most cognitive effort to exploration,
performance not much better than full task
Speaking Qualifier - Nov. 24, 2009 42
Future Work
• Effects of exploration autonomy– How is span-of-control affected by higher-level autonomy,
sliding autonomy?• Effects of perception/exploration task difficulty
– Do scaling effects hold in more realistic environments?• Scaling to larger team sizes (16, 24+)
– Do subtasks also have steep performance drop-offs?• Multi-operator interaction
– Can subtasks be distributed across operators?
Speaking Qualifier - Nov. 24, 2009 46
Methodology – Perceptual search
• If an autonomous path planner is used:– Covers a larger area than possible with a human operator (never pauses upon
arrival at a waypoint)
• If human generated trajectories are taken from the full task:– Pauses for waypoint completion– Pauses at locations where victims were found
• Solution: Use trajectories from the exploration condition– Contain pauses associated with waypoint arrival– Do not contain pauses for identifying victims.– Operators must be able to pause robots when they discover victims
Speaking Qualifier - Nov. 24, 2009 48
4 Robots 8 Robots 12 Robots20
25
30
35
40
45
fulltask exploration
Missions Assigned
F1,28 = 6.34
p < .018
Mis
sion
s as
sign
ed
Operator overload
Speaking Qualifier - Nov. 24, 2009 49
Average Path Length
4 Robots 8 Robots 12 Robots4
4.5
5
5.5
6
6.5
7
7.5
8
8.5
fulltask exploration
Aver
age
Path
Len
gth
(m)
Speaking Qualifier - Nov. 24, 2009 50
USARSim Validation Studies
• Synthetic video– Carpin, S., Stoyanov, T., Nevatia, Y., Lewis, M. and Wang, J. (2006a). Quantitative assessments of
USARSim accuracy". Proceedings of PerMIS 2006• Hokuyo laser range finder
– Carpin, S., Wang, J., Lewis, M., Birk, A., and Jacoff, A. (2005). High fidelity tools for rescue robotics: Results and perspectives, Robocup 2005 Symposium.
• Platform physics & behavior– Carpin, S., Lewis, M., Wang, J., Balakirsky, S. and Scrapper, C. (2006b). Bridging the gap between
simulation and reality in urban search and rescue. Robocup 2006: Robot Soccer World Cup X, Springer, Lecture Notes in Artificial Intelligence
– Lewis, M., Hughes, S., Wang, J., Koes, M. and Carpin, S., Validating USARsim for use in HRI research, Proceedings of the 49th Annual Meeting of the Human Factors and Ergonomics Society, Orlando, FL, 457-461, 2005.
– Pepper, C., Balakirsky, S. and Scrapper, C., Robot Simulation Physics Validation, Proceedings of PerMIS’07, 2007.
– Taylor, B., Balakirsky, S., Messina, E. and Quinn, R., Design and Validation of a Whegs Robot in USARSim, Proceedings of PerMIS’07.
– Zaratti, M., Fratarcangeli, M., and Iocchi, L., A 3D Simulator of Multiple Legged Robots based on USARSim. Robocup 2006: Robot Soccer World Cup X, Springer, LNAI, 2006.
[http://www.sourceforge.net/projects/usarsim]
Speaking Qualifier - Nov. 24, 2009 54
Conclusions
• Operators devote most cognitive effort to exploration– Automating exploration is necessary to increase operator
span-of-control when foraging– When operators are overloaded, performance can drop
dramatically
• Operators may be able to handle more than 12 robots when doing perceptual search alone– Most appropriate role for operator is sensor and
information fuser– Need control approaches that are observation, rather than
navigation, oriented
Speaking Qualifier - Nov. 24, 2009 55
Conclusions
• According to H1, full task performance drops more than expected as team size increases
• According to H2, exploration is largest contributor to operator workload
• Operator span-of-control limit between 8, 12• For larger teams, automate exploration