Workload and Situational Awarenessmanagement in UAV teams through interface

modelling

Antonio Sergio Ferreira

Faculdade de Engenharia da Universidade do Porto, Underwater Systems andTechnology Laboratory (USTL),

Rua Dr. Roberto Frias s/n, 4200-465 Porto, Portugal{asbf}@fe.up.pt

Abstract. The practical use of Unnamed Aerial Vehicles (UAV) to ex-tract the human element from dangerous situations is becoming a morefeasible scenario with every new iteration of technological advancement.Cost and implementation time reductions have led to the emergence ofvarious off-the-shelf solutions which allow a wide array of usage of thesesystems. However, on the most part, the development emphasis of thesesystems does not reside on usability quality or human factor effects. Thishampers the viability of theses systems in scenarios were human oper-ators must be focused on other duties other then vehicle control andsupervision, therefore a priority must be given to the control interface.Through the implementation of a Real-Time Strategy (RTS) game inter-face paradigm, in an already existing command and control framework,it was possible to develop a control and supervisory interface consolewhich achieved a decrease of workload in preliminary testing.

Keywords: HCI, HRT, UAV, Workload, Situational Awareness, Real-time strategy games

1 Introduction

With every new chapter of technological advancement, either in the hardwareand software sectors, it becomes easier or more cost-effective to develop and ap-ply Human-Robot Teams (HRT) in operation scenarios of dangerous nature. Awide array of scenarios, either simple Search and Rescue missions [1] or morecomplex military missions [2], are steadily becoming routine theatres of opera-tions for autonomous vehicles. Among the various types of automated vehiclesUAVs, through their ability to quickly cover large tracts of ground from an aerialperspective, have demonstrated be a valuable asset when aiding the activities ofhuman teams in a number of complex and demanding situations [3].

In order to do so these vehicles rely on complex control systems which allowthe human team member to take advantage of their capabilities, however therising complexity of said systems has led to an increasing need for more humanoperator per UAV platform. Nevertheless steps are being taken to allow future

unmanned vehicles systems to invert the operator-to-vehicle ratio so that oneoperator can control multiple vehicles connected through a decentralized network[4]. However this decentralization puts an increasingly high strain on the humanoperator’s workload [5] reducing the team’s overall performance [6] if steps aren’ttaken to deal with these variables.

There are various ways to cope with the workload reduction challenge. Itcan be achieved by an emphasis on further developing and extending the controlschema by creating frameworks which allow an adaptive level of automation [7][8] and dealing with the inner-workings of control algorithms or by addressinginterface usability issues, prioritizing the human user experience.

In order to achieve a low-entropy interface with the human operator, in astress filled situation, a great amount of attention must be given to the satura-tion point of information the operator can handle at one given moment [9][10].Conceiving the interface using a multi-modal approach is advantageous sinceit’s one way of avoiding the natural bottleneck that arises from using only onecommunication channel between the UAV and the operator [11].

Nevertheless, even with these approaches to reduce the operator workloadthere is another factor which must be taken into account when dealing withHuman-Computer Interaction (HCI). Every interface has a learning curve whichthe user must overcome in order to fully take advantage of its capabilities, there-fore it is safe to assume that the more familiar the interaction schema the lowerthe learning curve will be. It can than be assumed that video games could provea valuable asset when looking for familiar interface layouts and techniques. Onthis premise there have been various successful applications of game-based in-terfaces in HRT control [12][13]. For this paper the RTS game genera will beused as the basis for interface analysis. This type of game paradigm has alreadyshown promise in other research [14] surrounding autonomous vehicle control.

Fig. 1. Interface example of the Macbeth command and control system[21].

To deal with the emergent need to simplify the method of interaction withmore complex and capable UAVs the Neptus framework, developed by the Un-derwater Systems and Technology Laboratory at FEUP, has been extended fromit’s original role as a Command, Control, Communications and Intelligence (C4I)[19] for autonomous underwater vehicles (AUV) to also incorporate UAV controland supervision. It is upon this already established framework that the validityof an RTS paradigm will be tested. All development efforts were made so to min-imize the use of additional peripherals than the standard single monitor setup toon the one hand, reduce logistical system costs and on the other hand, increasethe number of operational scenarios where the console might be of viable use.

In order to evaluate both workload and situational awareness metrics specifictests will be employed. For workload evaluation the method used was the NASATask Load Index (NASA-TLX) questionnaire, already widely used in militaryand commercial aviation pilot workload tests [15][16]. For situational awarenessthe method used was the Situation Awareness Global Assessment Technique(SAGAT) also widely used, not only in the aviation field but also HRT testing[17][18]. Although time constraints and availability of certified humam operatorslimited the testing phase to only one account, the test proved promising resultsand will be augmented in future developments.

In Section 2 a quick overview of the framework used as basis for this paper isgiven followed by a conceptual description of the developed prototype in Section3. All the tests performed on the prototype are detailed in Section 4 and thesubsequent conclusions are presented in Section 5. In Section 6 an analysis ofthe envisioned future work is made.

2 Related Work

There already exist command and control frameworks which demonstrate anemphasis on the quality of the human interface, as seen in Fig. 1, with theintent of reducing workload in situations where one sole human operator mustcommand various autonomous vehicles. However the recurring standard interfaceparadigm is one of high complexity and high logistical and maintenance cost,even for the control of single autonomous platform, as demonstrated in Fig. 2.

One which attempts to reverse the aforementioned standard is the NeptusC4I framework which is a distributed framework for operations with networkedvehicles, systems, and human operators. Neptus supports all the phases of amission life cycle: planning, simulation, execution, and post-mission analysis.Moreover, it allows operators to plan and supervise missions concurrently [19].

The Neptus framework communicates with all autonomous vehicles thoughtthe IMC [20] communication protocol. This protocol, based on XML, augmentsthe frameworks compatibility with various kinds of autonomous vehicles by al-lowing a level of abstraction which promotes decoupled software development.

Furthermore, in order to minimize undesirable interdependencies through-out the framework’s architecture, the Neptus framework consists of a series ofindependent plugins surrounding a core section of functionalities.

Fig. 2. Interface example of the MOCU military system[2].

Nevertheless, Neptus encompasses a console building application which facil-itates the rapid creation of new operation consoles for new vehicles with new sen-sor suites, as well as the remodelling of old consoles for current vehicles (Fig. 3).It is one such console that ultimately will be conceived as a result of the workdeveloped.

Fig. 3. Example of a normal operational console provided by the Neptus C4I frame-work.

3 Prototype Development

Using the Neptus framework, development began towards the creation of a con-sole that would incorporate conceptual ideas implemented in modern RTS in-terfaces in order to further expand its usability by the human operator.

3.1 Conceptual Design Details

One of the major difficulties in this kind of endeavour is the simplification of thelarge amount of data deemed necessary for the safe operation and supervision ofUAV systems. The usually overwhelming quantities of numeric and textual datatend to lead to a multi-monitor solution which quickly raises the complexity ofthe interaction experience.In order to avoid this necessity a lot of informationfusion is needed.

The standard RTS game relies heavily on representative icons to convey in-formation to the player. Distinct icons can, on their own, quickly transmit largeamounts of information, that would otherwise require a large amount of textualrepresentation, just by small changes in size, orientation and draw rate. Withsimple icon modifications one can summarize a vast amount of the UAV’s rel-evant data. Coupled with this the RTS genera also employs color as a way toclearly indicate unit status and different levels of urgency. Both these elementscombined on their own can account for a great amount of information simpli-fication. However there are other aspects that can be extracted from the RTSexample which can prove invaluable to the overall console’s efficiency.

Fig. 4. Example of similarities in layouts between different RTS games interfaces.

Modern RTS games, although having different themes at their core, followa seemingly standardized formula when laying out their different interface com-ponents (Fig. 4). One can reason that this behaviour derived from the genera’svast years of experiments with different approaches which slowly converged intoa model that is now amply used by the majority of developers. Moreover, eventhe various interface components converged to form a pre-set of core interfaceelements that are re-hashed from game to game (mini-map, unit status, etc...).

These unofficial standards, combined with the ample size of the game market,led to an unintentional spread of these tendencies to a wide audience thus makingit a useful basis of development when aiming at a reduced interface learningcurve.

3.2 Architectural Details

Following the conceptual design, a layout for the console’s components was ide-alized. As shown in Fig. 5 the focus is to emulate the default RTS layout by

having a central large map region (section 1), a mini-map to aid the humanoperator to not lose context of other UAVs operating outside his view scope(section 2), a series of smaller additional panels which will house simplified nu-merical data (sections 3 through 5) and two final panels which will be used tohouse contextual information about other autonomous vehicles operating in thearea.

The central concept is to concentrate all the information with the highestpriority in the main panel (section 1) through use of icon disposition and color,relegating textual information to one of the support panels.

Fig. 5. Representation of the projected console layout.

Before the development of this new operation console started there was al-ready a pre-existent standard command console for UAV operations which servedas a benchmark in the later testing stage, in order to determine the effectivenessof the new design implementation.

3.3 Conceived Prototype

In Fig. 6 we can see the finalized operational version of the developed console.As expected, from the RTS paradigm influence, we have a large main map, aidedby a smaller mini-map, which provides a top-down view of the operation area.However the typical isometric point-of-view was not adopted due to renderinglimitations when attempting to model 3D environments. On a normal operationscenario it is complicated to acquire an up-to-date and accurate 3D model ofthe area so it was deemed that, as this stag,e a simple map-based representationwould be preferable.

Hovering over the map are icon indicators of the various stages of the UAV’sflight plan, from the pre-launch stage to the final runway taxi quickly giving theoperator a grasp of the mission’s current stage.

The mini-maps viewport feature allows the operator to quickly know whatpart of the operation theatre is being shown on the main panel, as well as whatare the current visual limits.

Fig. 6. Example of the prototype console developed.

Aiding the main viewing panel are a simple set of numerical data regardingvelocity and GPS indications, however the UAV’s actuator status is compliedand provided by the miniature segmented UAV model. Each section informs thehuman operator of its status by alternating color, providing the operator with afaster means of understanding the UAV’s overall condition.

At the far right of the console are two auxiliary panels one of which allows aquick visual reference of the all active autonomous vehicles’ altitudes and anotherwhich lists all detected autonomous vehicles in the operation area. Through thislist the operator can also switch between operating vehicles.

4 Tests and Results

In order to test the efficiency of the new proposed interface alterations, on reduc-ing the operator’s perceived workload and augmenting his situational awareness,a simulation was held at the USTL with one of its senior UAV operator.

The simulation asked for a team of 3 UAVs to accomplish a simple set ofsurveillance tasks over a pre-designated and already known area. One of theoperators was target of the evaluation while the other two were only consideredparticipants. Firstly the operator used the already existing Neptus UAV interfaceto execute his mission, afterwards the new developed interface was used.

4.1 Workload Analysis

At the end of the simulation the operator under evaluation was given a stan-dard NASA-TLX questionnaire to determine his perceived workload. The resultsobtained are shown in Fig. 7.

Fig. 7. Overall workload percentage comparison, segmented by NASA-TLX compo-nents.

4.2 Situational Awareness Analysis

During the simulation the operator would, from time to time, quickly be negatedaccess to the operation interface in order to answer a set of pre-determinedqueries with the intent of ascertaining how aware of his surrounding environmenthe was. The queries are as follows:

1. Point-out the position of each UAV currently active;2. Single-out the currently selected UAV;3. Determine the altitude of all active UAVs;4. Determine the altitude of the seleted UAV;5. Determine the speed of the seleted UAV;6. Determine the current UAVs waypoint;7. Determine the current UAVs ID.

A total of ten interruptions were made and the results obtained are shownin Fig. 8.

These preliminary results show promise as it seems accurate to extrapolatethat the modifications made on the command interface, using the RTS paradigm,have lowered the operators workload and, at the same time, risen his situationalawareness during the mission’s execution.

5 Conclusions

Throughout this paper references were made to the growing importance of UAVsystems, paying special attention to their valuable application in real world sce-narios. It was then presented the concepts behind a possible solution to be imple-mented into a pre-existent framework with the ultimate goal of managing UAVworkload a mission scenario and the details around this solution were presentedand discussed.

Fig. 8. Overall situational awareness percentage comparison, throughout a series ofpre-determined queries.

The C4I operation console, ultimately created based on the presented solu-tion, enabled the reduction of the workload felt by the operator while at thesame time increasing his situational awareness.

6 Future Work

Although promising these type of results are, by their nature, subjective to thehuman operator’s sense of workload and own experience of operation procedure.This means that further testing must be performed in order to ascertain if theoverall improvements detected prevail in a wide variety of human operators.Furthermore, a closer look into the level of situational awareness gained must betaken by providing more challenging and complex scenarios to work with, whileexpanding the set of tasks to be evaluated.

References

1. Kadous, M.W., Sheh, R.K.-M., Sammut, C.: Controlling Heterogeneous Semi-autonomous Rescue Robot Teams. 2006 IEEE International Conference on Systems,Man and Cybernetics. pp. 3204-3209. IEEE (2006).

2. Powell, D.: Multi-robot operator control unit. Proceedings of SPIE. p. 62301N-62301N-8. SPIE (2006).

3. Jones, G., Berthouze, N., Bielski, R., Julier, S.: Towards a situated, multimodalinterface for multiple UAV control. 2010 IEEE International Conference on Roboticsand Automation. pp. 1739-1744. IEEE (2010).

4. Cummings, M.L., Clare, A., Hart, C.: The Role of Human-Automation Consensus inMultiple Unmanned Vehicle Scheduling. Human Factors: The Journal of the HumanFactors and Ergonomics Society. 52, 17-27 (2010).

5. Prewett, M.S., Johnson, R.C., Saboe, K.N., Elliott, L.R., Coovert, M.D.: Managingworkload in humanrobot interaction: A review of empirical studies. Computers inHuman Behavior. 26, 840-856 (2010).

6. Crandall, J.W., Cummings, M.L.: Developing performance metrics for the supervi-sory control of multiple robots. Proceeding of the ACMIEEE international confer-ence on Humanrobot interaction HRI 07. 33 (2007).

7. Bocaniala, C.D., Sastry, V.V.S.S.: On enhanced situational awareness models forUnmanned Aerial Systems. 2010 IEEE Aerospace Conference. pp. 1-14. IEEE(2010).

8. de Brun, M.L., Moffitt, V.Z., Franke, J.L., Yiantsios, D., Housten, T., Hughes, A.,Fouse, S., Housten, D.: Mixed-initiative adjustable autonomy for human/unmannedsystem teaming. AUVSI Unmanned Systems North America Conference (2008).

9. KABER, D., WRIGHT, M., SHEIKNAINAR, M.: Investigation of multi-modal in-terface features for adaptive automation of a humanrobot system. InternationalJournal of Human-Computer Studies. 64, 527-540 (2006).

10. Yanco, H.A., Drury, J.L., Scholtz, J.: Beyond Usability Evaluation: Analysis ofHuman-Robot Interaction at a Major Robotics Competition. Human-Computer In-teraction. 19, 117-149 (2004).

11. Maza, I., Caballero, F., Molina, R., Pea, N., Ollero, A.: Multimodal Interface Tech-nologies for UAV Ground Control Stations. Journal of intelligent and robotic sys-tems. 57, 371391 (2010).

12. Hassell, A.J., Smith, P., Stratton, D.: An evaluation framework for videogamebased tasking of remote vehicles. Proceedings of the 4th Australasian conference onInteractive entertainment. p. 10. RMIT University (2007).

13. Aubert, T., Corjon, J., Gautreault, F., Laurent, M.: Improving situation awarenessof a single human operator interacting with multiple unmanned vehicles: first results.26-27 (2010).

14. Jones, H., Snyder, M.: Supervisory control of multiple robots based on a real-time strategy game interaction paradigm. 2001 IEEE International Conference onSystems, Man and Cybernetics. e-Systems and e-Man for Cybernetics in Cyberspace(Cat.No.01CH37236). pp. 383-388. IEEE (2001).

15. Whetten, J.M., Goodrich, M.A.: Beyond robot fan-out: Towards multi-operatorsupervisory control. 2010 IEEE International Conference on Systems, Man and Cy-bernetics. pp. 2008-2015. IEEE (2010).

16. Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., Goodrich,M.: Common metrics for human-robot interaction. Proceedings of the 1st ACMSIGCHI/SIGART conference on Human-robot interaction. pp. 3340. ACM, NewYork, New York, USA (2006).

17. Adams, M.J., Tenney, Y.J., Pew, R.W.: Situation awareness and the cognitivemanagement of complex systems. Human Factors The Journal of the Human Factorsand Ergonomics Society. 37, 85-104 (1995).

18. Stanton, N.: Handbook of Human Factors and Ergonomics Methods. CRC Press(2004).

19. Dias, P., Pinto, J., Gonalves, R., Gonalves, G., Sousa, J.: Neptus, command andcontrol infrastructure for heterogeneous teams of autonomous vehicles. ICRA - IEEEInternational Conference on Robotics and Automation. 2768-2769 (2007).

20. Martins, R., Dias, P.S., Marques, E.R.B., Pinto, J., Sousa, J.B., Pereira, F.L.:IMC: A communication protocol for networked vehicles and sensors. OCEANS 2009-EUROPE. pp. 1-6. IEEE (2009).

21. Simmons, R., Apfelbaum, D., Fox, D., Goldman, R.P., Haigh, K.Z., Musliner, D.J.,Pelican, M., Thrun, S.: Coordinated deployment of multiple, heterogeneous robots.Proceedings. 2000 IEEE/RSJ International Conference on Intelligent Robots andSystems (IROS 2000) (Cat. No.00CH37113). pp. 2254-2260. IEEE (2000).