design and evaluation of intelligent adaptive operator

RTO-HFM-135 8 - 1

Design and Evaluation of Intelligent Adaptive Operator Interfaces for the Control of Multiple UAVs

Ming Hou Robert D. Kobierski Defence Research & Development Canada – Toronto CMC Electronics Inc.

1133 Sheppard Ave West 415 Legget Drive Toronto, ON, Canada M3M 3B9 Ottawa, ON, Canada K2K 2B2

[email protected] [email protected]

ABSTRACT

An increasingly important issue for multi-Uninhabited Military Vehicle (UMV) control is the management of massive amounts of information to support effective decision making. Feedback from UMV operators indicates that improvements in operator interfaces would reap significant gains in system performance and effectiveness. Various levels of automation have been suggested to address the problem including Intelligent/Adaptive Interfaces (IAIs) for decision support. Augmenting UMV control stations with automation function groups, IAIs are intended to manage information dynamically and provide the right information to the right people, at the right time, to support effective decision-making. The work reported here is the result of the last two phases of a multi-year project conducted by Defence Research and Development Canada. This study investigated the efficacy of IAIs in a multi- Uninhabited Aerial Vehicle (UAV) scenario with the IAI modelled as part of the UAV tactical workstations found in a maritime patrol aircraft. A performance model was developed in the first phase of the project to compare the difference in mission activities with and without automation as reflected in task conflict frequency and task completion time. A prototype IAI experimental environment has been implemented in the second phase for a human-in-the-loop empirical investigation conducted in the third phase. Both simulation and experiment results revealed that the control of multiple UAVs is a cognitively complex task with high workload. With the augmentation of automation agents, IAIs facilitated a significant reduction in workload and an improvement in situation awareness. Operators could continue working under high time pressure, resulting in reduced completion time for critical tasks when comparing to conventional interfaces.

1.0 INTRODUCTION AND MOTIVATION

The deployment and control of Uninhabited Military Vehicles (UMVs) generates an enormous amount of data that will become even more complex as the communications between air, sea, and ground have more channels for joint operations. As the quantity and variety of those data increase, the workload of UMV operators is likely to increase exponentially, thus imposing severe constraints on personnel conducting these missions. One way to reduce operator demands is to convert those data into the right kind of information and automatically disseminate it to the right decision makers. Another way is to look for opportunities to limit the complexity of tasks that humans perform in controlling UMVs, and a third seeks to limit the number of tasks to be performed.

Feedback from the operation of UMVs indicates that there is a need for improvement in the operator interfaces of these emerging systems. This applies both to effective control of UMVs and to the management

8 - 2 RTO-HFM-135

NATO UNCLASSIFIED

of data, including converting those data to information and efficiently disseminating it to appropriate operators. The level of automation (intelligent and adaptive software) to be applied to the decision-making processes is a key factor for both tactical commanders and UMV system managers [1] [2] [3] [4]. As a result, supporting technologies that combine both operators and automation to satisfy mission requirements need to be investigated.

Intelligent Adaptive Interfaces (IAIs) are one of such technologies that are intended to reduce the impact of operator interface complexity and operator workload. IAIs are human-machine interfaces that would improve the efficiency, effectiveness, and naturalness of human-machine interaction by acting adaptively and proactively to external events based on internal mission requirements [5]. Specifically in the context of UMV control, an IAI is a workstation graphical user interface (GUI) driven by software agents that support the decision-making and action requirements of operators under different levels of workload and task complexity. The IAI manifests itself by presenting the right information or action sequence proposals, or perform actions, at the right time.

Having realized the key issue of operator interfaces for mission success, Defence Research & Development Canada (DRDC) initiated a multi-year project for the development and evaluation of IAIs for multiple Uninhabited Aerial Vehicle (UAV) control. The aim of this project was to develop, demonstrate, and prioritize enabling technologies that can be applied to advanced operator interfaces that will support reduced manning and enhanced performance in complex military systems, particularly multiple UAV control from an airborne platform. This project laid the foundation for the production of preliminary design guidelines for IAIs.

The DRDC IAI project had three phases: IAI concept development, interface prototyping, and experimentation. Phase I of the project involved in an analysis of UAV operations in a mission scenario which was in support of counter-terrorism activities. The selected environment involved operations with the IAI modelled as part of the UAV tactical workstations of a modernized Canadian CP140 maritime patrol aircraft. In the scenario, the augmented CP140 crew took over the UAV operation in the role of UAV Pilot (UP), UAV Sensor Operator (UO), and Tactical Navigator (TN) in the tactical compartment of the aircraft. The analytical results were used to develop a performance model that was then implemented in an integrated performance modelling environment. The model was run in two modes: one assuming the operators used a conventional interface to control the UAVs and the second assuming interface automation used an IAI. The difference between mission activities with and without automation was reflected in the time to complete critical task sequences and in frequency of task conflict. The simulation revealed that the use of a control console, which incorporates an IAI mode, permitted operators to continue working under high time pressure, resulting in upper level goals being achieved in reduced time. The detailed information about the work done in the first phase is included in Hou & Kobierski [6] [7].

This paper reports the result and its implications of Phase II and III of this DRDC IAI project. The work in these two phases focused on the design and implementation of IAI prototype interfaces and experimentation itself for the investigation of IAI efficacy.

2.0 IAI EXPERIMENTAL ENVIRONMENT AND MISSION SCENARIO

An experimental synthetic environment (SE) was designed and developed using the NATO STANAG 4586 interface software protocol. Consistent with the UAV crew positions used in the first project phase, the SE had three control consoles replicating CP140 tactical compartment multifunction workstations. The workstations

IAIs FOR THE CONTROL OF MULTIPLE UAVS

RTO-HFM-135 8 - 3

were designed to communicate with virtual UAVs through fully functional real world software interfaces. Each of them had a set of appropriate displays and controls for the UP, UO, and TN. The experimental environment also had the ability to integrated video and audio data collection thus providing for empirical assessments of IAI concepts developed in the first phase.

2.1 Physical Layout and Organization

It was assumed that in an operational subsystem, the UAV operators would occupy the three rear positions of the CP140’s tactical compartment, as illustrated in Figure 1. The experimental environment was then built to dimensionally match the CP140 tactical workstations, and the overall configuration is shown in Figures 2 and 3. Each member of the UAV crew (i.e., UP, UO, and TN) was provided with their own workstation, which consisted of a main display screen, a keyboard, a programmable entry panel (PEP), a trackball/mouse, and a joystick (for UP and UO only). The UP and the UO were seated next to each other and shared the same console. The UP and the UO also shared a display screen (i.e., the middle display in Figure 3), which was mounted between their individual workstation displays. The shared display was a modified version of the TN’s TACPLOT that showed various contacts and their associated track numbers.

Figure 4 shows TN’s primary display, which was designed for communicating tactical information. The display area was a large TACPLOT that the TN used to supervise and manage the tactical situation and it was consistent with TN’s main role as coordinator of the CP140 UAV crew.

Figure 5 shows the UP’s primary display, which was provids the information necessary for piloting UAVs. The basic layout allowed operators to display a mini TACPLOT and pilot camera view for up to two UAVs. Note that all pilot camera views were superimposed with heads-up-display (HUD) style symbology to communicate critical flight data. Since some UAV types do not have a pilot camera, a solid background was used to replace the pilot camera image allowing the presentation of the same HUD-style symbology.

Figure 1: CP140 Tactical Compartment Layout with TN, UP and UO Positions

8 - 4 RTO-HFM-135

NATO UNCLASSIFIED

Figure 2: Experimental Environment Showing Three UAV Control Workstations and NAVCOM Workstation (to the right)

Figure 3: UP and UO Positioned at the Aft Rearward-Facing Workstations


RTO-HFM-135 8 - 5

Figure 6 shows the UO’s primary display, which was designed for providing the information necessary to manage and extract information from a large number of sensors. The main display area allowed operators a great deal of flexibility in creating layouts for the display of sensor data and in switching between those layouts. Each sensor video was superimposed with information that allowed sensor operators to easily comprehend the instantaneous direction the turret was pointing, with respect to the current heading of the UAV, as well as the sensor elevation angle and zoom setting.

2.2 IAI Software Agent

The IAI agents were functional components of the UAV control SE developed for this research. They supported the experiment participants in accomplishing the assigned mission tasks of the experiment by providing decision support to the crew and by taking over certain high workload crew tasks.

In order to offer help, the IAI agents were designed to follow a defined sequence as follows:

Step 1: Gather status information about all active UAVs, the tracks pertinent to those UAVs and the current display configuration.

Figure 4: Primary Display for TN

8 - 6 RTO-HFM-135

NATO UNCLASSIFIED

Figure 5: Primary Display for UP

Figure 6: Primary Display for UO


RTO-HFM-135 8 - 7

Step 2: Analyse the information with respect to pre-defined rules and determine which events have occurred.

Step 3: Prioritize the events with respect to pre-defined prioritization rules. Step 4: Execute pre-defined tasks for each identified event following the order of prioritization.

Depending on the results of the analyses, IAI agents would actively support the operators with the following tasks:

• Route Planning. If a UAV was turned in to investigate an unknown or hostile contact, the agent would compute the most direct route and activate that route for the UAV. The allocation of tracks to UAVs was based on a search for closest unknown or hostile contact. Additional logic ensured that no more than one UAV could be engaged on a single unknown contact, but more than one UAV could be engaged on hostile contacts.

• Route Following. The agent would pilot the UAV safely on the active route. This task also included flight altitude and speed management as well as self-preservation in close proximity to the track. In addition, the agent would enter an orbital flight pattern around the track once the UAV had reached sensor identification range.

• Screen Management. The shared TACPLOT was managed by the agent whenever new high-priority events would occur. This included panning the TACPLOT to a location of interest and zooming in and out.

• Inter-crew Communications. All observations pertaining to UAVs in relationship to tracks were reported by the agent to the crew via the IAI message window. In that way the crew’s need for voice communication indicating or confirming theses observations was significantly reduced.

• Sensor Management. Once a UAV was close enough to the track to engage the EO sensor, the agent would take over the management of the sensor. This task included the pointing of the sensor and the establishment of stable lock on the moving target once the track was within visual range.

• Data Link Monitoring. The agent monitored the flight pattern and other vitals signs of the UAV in order to determine whether the data link was still working. If not, the agent would immediately inform the crew about the lost data link.

Therefore, there were six software agents were designed and implemented in the IAI prototype interfaces as multi-agent subsystems.

To allow the agents to communicate events and actions to the crew, the IAI GUI was designed to allow for IAI messages to be displayed in the primary display thus attaining the crew’s attention faster and more reliably. As an example, Figure 7 illustrates an IAI message window, which was located on the operators’ primary display (also see small black windows at the bottom left corner of the displays in Figures 4, 5 and 6). The IAI message window held a list of all active UAVs, the allocated track and additional information

8 - 8 RTO-HFM-135

NATO UNCLASSIFIED

pertaining to the IAI agent’s support for this UAV. In addition, textual readouts were added to the UAV icon on the TACPLOTs, as illustrated in Figure 8. These readouts identified the contact ID designated by the IAI agents and whether the IAI agents is currently exercising UAV flight control (“P”) and/or sensor geotracking (“S”).

Figure 7: IAI Message Window

Figure 8: IAI Readouts in TACPLOT

2.3 The Mission Situation

In order to evaluate IAI prototype interfaces, a mission scenario was developed for carrying out a counter-terrorism mission. The scenario was set in the year 2011 when the Commonwealth of Nations has chosen St John’s, Newfoundland, Canada as the site for the bi-annual Commonwealth Heads of Government Meeting (CHOGM). The Canadian Forces (CF) provided security for the meeting as they had at many international meetings. At 1745 hrs August 11th, 2011, British Intelligence relayed information about a Lethal Medium Range UAV (poor man’s cruise missile), which might be a threat to the CHOGM. This device could be launched from within a van-sized steel container. The intelligence groups suspected that the Lethal UAV would be launched from a boat that might be as far as 240 nm away. An addendum to the Lethal UAV warning made reference to recent reports that the group suspected of fielding the weapon had obtained a quantity of plutonium from a nuclear power plant. A UAV carrying a plutonium “dirty bomb” would cause many casualties and render the targeted region inaccessible for years.

In the meantime, there was an ongoing fisheries patrol southeast of St. John’s, and approximately 200 vessels had been plotted in the vicinity of the nose and tail of the Grand Banks, Canada. A Canadian maritime frigate HMCS Halifax was on-scene with two Vertical Take-Off UAVs (VTUAV) and a Maritime Helicopter (MH). One CP140 patrol aircraft equipped with 16 Mini UAVs and its own sensor suite was also overhead. Figure 9 illustrates the situation off the southeast coast of Cape Race, Newfoundland.

The scenario began at 1800 hrs, after the CP140 crew had received information that a terrorist threat to the CHOGM was possible and they were re-tasked to search for a vessel that was carrying a launch container (approximately 10 ft x 8 ft x 20 ft). Reports had suggested that the threat might come from a trawler-sized vessel. They were provided with a VTUAV from HMCS Halifax. Once the VTUAV cleared its mother ship, HMCS Halifax made ready and launched the MH. The mission was to investigate another concentration of vessels to the south. The ship’s crew knew that recovery of VTUAV 1 would be necessary at approximately the same time that the MH returned, but the situation dictated that two VTUAVs and the MH were airborne.


RTO-HFM-135 8 - 9

At 1843 hrs, contact was lost with VTUAV 1 as it approached a vessel under investigation. The CP140 immediately launched three Mini UAVs over the contact and warned other airborne units to avoid the possible threat. The Mini UAVs investigated the vessel, and at the same time HMCS Halifax made best possible speed to the same location. VTUAV 2 was also directed towards the suspicious boat and control was passed to the CP140 crew.

At approximately 1850 hrs, a Mini UAV transmitted an image of men working on the foc’sle of the trawler. These individuals had exposed a large storage container. The CP140 continued to covertly observe through the EO sensor of VTUAV 2 and Mini UAVs as the container was opened to expose a Jet Assist Take-Off (JATO) UAV. Minutes later, the lethal UAV was launched. Two CF18s were ordered to attack the now identified terrorist boat. Assisted by a laser UAV controlled from the CP140 the terrorist boat was destroyed prior to launching a second lethal UAV. At 1900 hrs, the experimental scenario ended, although the hypothetical mission was still ongoing. The CP140 crew initiated a search for the Lethal UAV, which was tracking randomly towards St John’s.

Group of Fishing Vessels Approximately 50 boats



CP-140

HMCS Halifax VTUAV 2

VTUAV 1

MH 0 50 100

Statute Miles

Time 1800 hrs

MALE UAV

MALE UAV Patrol Area

CP140 Patrol Area

CPF Patrol Area

Figure 9: Grand Banks Overview at 1800 Hrs

Canadian CP140 Aircraft

8 - 10 RTO-HFM-135

NATO UNCLASSIFIED

3.0 EXPERIMENTAL DESIGN

3.1 Participants

As described in Section 2.1, the roles of UAV operators as experiment participants in this study were assigned as:

• Tactical Navigator (TN). The TN was the mission commander, and was responsible for managing the use of all available resources to accomplish the mission goals.

• UAV Pilot (UP). The UP was responsible the deployment, management, and control of individual UAVs.

• UAV Sensor Operator (UO). The UO was responsible for the selection and management of the simulated Electro-Optic (EO) sensors and the interpretation of data from these sensors.

Thus, eight UAV crews, each comprised of three members were recruited as volunteers from the CP140 community at Canadian Forces Bases (CFB) Comox and CFB Greenwood. All participants were male and ranged from 26 to 52 years of age. All had operational experience with the CP140. A member of each crew was assigned the role of UP, another member the role of UO, and the final member the role of TN. This assignment of roles was based on the amount of experience in the aforementioned positions, with crewmembers being assigned to the position with which they had the most experience. The participants used for this experiment, all of whom were “fit to fly”, were considered medically suitable for operator interface experiments in this simulated CP140 tactical compartment. The UAV crews were supplemented by an experimental staff member who played the role of Navigation Communicator (NAVCOM). As the experimental scenario unfolded, the NAVCOM stimulated the subject crew (i.e., UP, UO, and TN) with “taskings” from the Maritime Operations Centre (MOC) at Maritime Forces Atlantic, in Halifax, Canada.

3.2 Independent Variables

A 3 x 3 x 2 (Operator Workload: mission part 1 vs. mission part 2 vs. mission part 3; Operator Position Complexity: UP vs. UO vs. TN; IAI Condition: OFF vs. ON) mixed factor design was used in the experimentation phase. The three levels of Position Complexity correspond to the three crew positions (i.e., UP, UO, and TN). The three levels of Operator Workload correspond to the three parts of the mission scenario. Part 1 of the scenario was designed to induce the lowest workload given that it involved the control of only one VTUAV with the task of prosecuting only one contact. Part 2 of the scenario was designed to produce moderate workload given that it involved the control of two UAVs (i.e., one VTUAV and one mini UAV) with the task of prosecuting two contacts. Part 3 of the scenario produced the highest workload given that it involved the control of up to five UAVs (i.e., one VTUAV, three mini UAVs and one laser designator UAV) with the task of prosecuting three contacts while keeping an “eyes-on” a fourth contact. The two levels of IAI Condition refer to whether the crew was using either the standard interface (i.e., IAI OFF) or the standard interface augmented with IAI (i.e., IAI ON). The IAI for each crewmember was tailored to suit their needs. Both Operator Workload and IAI Condition were within-subjects factors, meaning that participants were tested under all levels of those factors, whereas Position Complexity was a between-subjects factor (i.e., crewmembers remained at one position throughout the experiment). A depiction of this experimental design can be seen in Figure 10 below.


RTO-HFM-135 8 - 11

3.3 Dependent Variables

Both objective and subjective measures were used to index each crewmember’s performance under (a) the three levels of position complexity, (b) the three levels of operator workload condition, and (c) both levels of IAI condition.

3.3.1 Objective Measures

Completion Time for Critical Task Sequences (CTSs) As a primary measure, it was the time to complete all the CTSs. The CTSs were defined a priori using a Hierarchical Goal Analysis (HGA) conducted in the first phase of the project [8], but were limited to tasks that had objective and clearly observable start and end points. They were further constrained by the scenario itself, in that the tasks had to be performed by each crew in the same manner so as to avoid excessive variability in the data across the different crews.

Percentage of CTS Shedding This measure was designed to compliment the CTS completion time measure in that it highlights deficiencies in the strategy of quickly and accurately completing only a few critical tasks at the cost of not attending to other critical tasks under high workload conditions. Given that only successfully completed CTSs were included in the completion time analysis, this strategy of shedding tasks would not be captured in these data. That is, crews that adopted this strategy would appear to have performed quite well because their completion times would have been quite low. The reality is, however, that their performance might have been quite poor; depending on the number of tasks they shed (i.e., did not complete). The percentage of CTS shedding measure highlighted the number of incomplete tasks.

The percentage of CTS shed was calculated by dividing the number of valid (i.e., unaffected by extraneous variables) incomplete sub-CTSs by the total number of sub-CTSs and then multiplying this result by 100. Unlike the CTS task completion time measure, the CTS shedding measure was only calculated for all CTSs and not for CTS subset. It was not necessary to describe this measure in terms of a CTS subset as it was

IAI ON

IAI OFF UP UO TN

Low

Med

High

Level of Position Complexity (Task Domain)

Level of Automation (Interface Domain)

Level of Temporal Workload (Operator Domain)

Figure 10: Depiction of the 3 x 3 x 2 Mixed Factor Experimental Design

8 - 12 RTO-HFM-135

NATO UNCLASSIFIED

calculated in terms of a percentage and is therefore not susceptible to the statistical issues associated with (a) an unequal number of CTSs across mission parts and (b) different tasks across mission parts. As with the CTS completion time measure, the CTS task shedding measure was only be analyzed for parts 2 and 3 of the mission scenario.

Route Trajectory Score It was an assessment of the effectiveness of the participants to fly the best routes to investigate all potential contact was determined. It was calculated independently for each critical UAV in each of the six mission parts. There was one critical UAV in mission part one, two critical UAVs in mission part two and four critical UAVs in mission part three. These seven critical UAV trajectory scores were calculated for both IAI conditions (i.e., IAI ON and IAI OFF), therefore yielding a total of fourteen route trajectory scores for each crew. The route trajectory score for each critical UAV was calculated by determining the average difference in distance between the actual trajectory and the optimal trajectory. As such, better performance is indicated by a lower route trajectory score. With that in mind, a correction factor was applied to each route trajectory score, which penalized crews for violating a ½ nm “no-fly” airspace that surrounded each contact. This correction factor increased the route trajectory score proportionally with the amount of time spent violating the ½ nm airspace.

Airspace Violation Time It was the amount of time that the UAVs flew within ½ nm of a vessel. This “stand-off” zone was briefed as a no-fly area and time spent within this area was counted against the crew. The measure is the average amount of time (in seconds) that each crew spent flying their critical UAVs in the ½ nm radius that surrounded each contact. This measure was designed to provide a detailed perspective on the crews’ navigational errors by highlighting the temporal magnitude of each airspace violation.

3.3.2 Subjective Measures

All subjective measures reported herein were collected from a questionnaire that was administered after each of the six mission parts. These questionnaires were designed to measure the crew’s subjective assessment of their:

• situation awareness (SA) of mission activities

• workload associated with the need to perform

• overall workload associated with mission tasks

A separate questionnaire was also given to the NAVCOM after each of the six mission segments where he or she rated the performance of the each member of the crew separately and the performance of the crew as a whole. The NAVCOM was an essential member of the crew in addition to being both a SME and an important part of the experimental staff. These characteristics allowed for a fair, accurate and independent assessment of the crew’s performance.

3.4 Mission Versions

Each crew was tested under both IAI Conditions under each of the three Operator Workload levels. As such, each crew experienced each of the three mission parts twice (once with IAI OFF and once with IAI ON). In order to avoid having the crew’s memory from their first experience with a mission part influence their second exposure to a mission part (which would most likely happen if the two mission parts were identical), a second version of each mission part was created by taking the original mission layout and rotating its mirror image.


RTO-HFM-135 8 - 13

The track identification numbers for each contact were changed in order to further increase the disparity between the two versions of the mission.

3.5 Counterbalancing

UAV crews participated in six experimental missions in total. Part 1 of the mission always occurred first, once with IAI OFF and once with IAI ON. Part 2 of the mission always occurred second, once with IAI OFF and once with IAI ON. Part 3 of the mission always occurred last, once with IAI OFF and once with IAI ON. So as to prevent practice effects from contaminating the data (i.e., improved crew performance due to increasing familiarity with the user interface and UAV flight dynamics), IAI Condition was counterbalanced across crews. That is, half of the crews received the IAI OFF condition first and the IAI ON condition second. This order was reversed for the other half of the crews. Additionally, even though every attempt was made to equate the original version of a mission part with its modified version, there remains the possibility that they were not of equal difficulty. As such, mission version was also counterbalanced across crews. That is, half of the crews received the original mission parts first and the modified version second. This order was reversed for the other half of the crews. As such, four crews were required for a complete experimental counterbalance. Therefore, there were eight crews participated in the experiment.

3.6 Procedure

Following a brief introduction to each member of the experimental staff, the crews were given a ten minute overview of the general purpose of the experiment and were given the opportunity to ask questions. The crews were then given an approximately two hour training session on how to use the workstation interface. Immediately following this training session, the crews were then given a practice scenario where they were tasked to investigate a single contact using a VTUAV. The crews practiced until they were familiar enough with their workstation functions that they could identify this contact both quickly and effectively.

After the practice, the crews were then given a 20-minute briefing by the NAVCOM, whose primary role was to act as the liaison between the experimental staff and each crew. This detailed briefing covered various mission aspects including the mission scenario, the number of mini UAVs they had at their disposal and, perhaps most importantly, the key visual features that distinguished the terrorist vessel from other unknown vessels (i.e., hydraulic lifts on the side of a ship container and doors that opened horizontally instead of vertically).

Once questions regarding the information covered in this briefing were answered, the crews began the first of six experimental sessions. The first experimental session typically took about 20 minutes to complete. Immediately following this session, each crewmember was asked to complete the subjective questionnaire that asked them to rate various aspects of their situational awareness, performance, confidence, and workload.

Right after completion of the questionnaire, the crews began the second of six experimental sessions, which also took about 20 minutes to complete and were again given a questionnaire. The first day of testing terminated once the questionnaire was completed.

On the second day of testing, the crews completed the remaining four experimental missions and the associated questionnaires. The crews typically completed two of the experimental sessions in the morning and two in the afternoon. The third and fourth sessions each took approximately 25 minutes to complete whereas the fifth and sixth sessions required about 30 minutes. Upon completion of the sixth (and final) session and questionnaire, the crews were asked to complete a usability questionnaire regarding the realism of

8 - 14 RTO-HFM-135

NATO UNCLASSIFIED

the scenario and the efficacy of the operator-machine interface (e.g., the TACPLOT, the PEP). Following completion of the usability questionnaire, the crews were debriefed by the experimental staff and the critical aspects of the experiment were revealed. The crews were also reminded not to divulge any of this information to other military personnel at CFB Comox or CFB Greenwood as it could contaminate the data if future crews knew the purpose of the experiment prior to participating in it.

4 EXPERIMENT FINDINGS

Analyses were performed for eight above dependent measures on eight crews with two completed sets of counterbalanced data. All data were subjected to a paired and three paired samples t-test comparing performance across both levels of IAI Condition (ON vs. OFF), three levels of Operator Workload (mission part 1 vs. 2 vs. 3), and three level of Position Complexity (UP vs. UO vs. TN), respectively. These eight measures are:

• completion time for various CTS (lower CTS completion time is better)

• percentage of CTS shedding (fewer sequences shed is better)

• corrected route trajectory score (values <½ nm/min represent a good trajectory with little wasted time)

• time that the UAVs flew within ½ nm of a vessel (lower total time violations the better)

• SA for various tasks (higher score is better)

• workload associated with the need to perform (low ratings indicated higher performance)

• overall workload associated with mission tasks (low ratings are better)

• NAVCOM Assessment of Crew Performance (higher ratings are better)

The results and associated 95% confidence intervals for both levels of IAI Condition, two or three levels of Operator Workload, and three levels of Position Complexity are shown in Figures 11 to 19. Although CTS were determined a priori for mission part 1, they were not included in this analysis as they were of little interest due to the minimal workload associated with this part of the scenario. As such, only CTS and other measures associated with mission parts 2 and 3 will be discussed.

4.1 Critical Task Sequence Completion Time

The CTS data were subjected to a 2 (IAI Condition: ON vs. OFF) x 2 (Operator Workload: Mission Part 2 vs. Mission Part 3) repeated measure of analysis of variance (ANOVA). Crews were faster at completing the CTSs when the IAI was ON (83.7 sec combined value) than when it was OFF (104.8 sec combined value). This difference, however, only approached significance, t(7) = 1.77, p < 0.13. The percentage improvement in CTS completion times with IAI selected ON was essentially the same regardless of the Mission Part. The Mean CTS Completion data is shown in Figure 11.


RTO-HFM-135 8 - 15

4.2 Percentage of CTS Shedding

The CTS task shedding data were subjected to a 2 (IAI Condition: ON vs. OFF) x 2 (Operator Workload: Mission Part 2 vs. Mission Part 3) repeated measure of ANOVA. These data are shown in Figure 12 and it can be seen that crews shed a significantly smaller percentage of tasks in mission part 2 than in mission part 3, t(7) = 8.22, p < 0.001. It is also noted that the number of tasks were shed in both parts 2 and 3 was not significantly different between IAI ON and IAI OFF.

Mean CTS Completion Time as a Function of Mission Part and IAI Condition

0

20

40

60

80

100

120

140

2 3Mission Part

Co

mp

leti

on

Tim

e (s

ec.) IAI OFF IAI ON

Figure 11: Mean Completion Time and 95% Confidence Intervals for all CTSs

as a Function of Mission Part and IAI Condition

Percentage of CTSs Shed as a Function of Mission Part and IAI Condition

0

5

10

15

20

25

30

35

2 3

Mission Part

CT

Ss

Sh

ed (

%)

IAI OFF IAI ON

Figure 12: Percentage of CTSs Shed and 95% Confidence Intervals as a Function of Mission Part and IAI Condition

8 - 16 RTO-HFM-135

NATO UNCLASSIFIED

4.3 Route Trajectory Score

Figure 13 shows an example of the Route Trajectory Score, which is a sample plot from Crew 7 performance during Part 3 of IAI OFF. The data associated with the mission part are contained in the bottom left of the plot. In addition to information such as the scale of the drawing and the numbers of the UAV and boat being investigated, each plot data set contains the relevant results for the mission part. These results include:

• time period (if any) that the UAV was flown within the ½ nm stand-off (no fly) circle from each boat (note that in the sample plot what appears to be a gross violation of the ½ nm stand-off circle was in fact only a minor violation because although the UAV flight path stays fixed, the stand-off circle moved with the boat);

• the mission time at which the UAV first approach to within 3 nm of the boat. This was used for subsequent calculations of time to complete critical task sequences; and

• the trajectory score, which is the average number of nautical miles unused (per minute) because the UAV was not flown at high speed directly towards the closest unknown boat.

Mission Time (minutes)

3 nm Range Ring (at time 45 min 13 sec)

Unknown Vessel

0.5 nm Range Ring (at time 53 min)

Portion of UAV Route used for Position Error Calculation

UAV Symbol

Position Error Calculation Construction Lines

Loiter Pattern

Figure 13: Sample Trajectory Plot


RTO-HFM-135 8 - 17

The corrected route trajectory score data were subjected to a 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) repeated measure of ANOVA. The corrected route trajectory score was significantly higher (a lower number is better) when IAI was OFF (0.69 nm per min combined value) than when it was ON (0.40 nm per min combined value), t(7) = 4.63, p < 0.005. This indicates that crews were better at maintaining an optimal flight path when IAI was ON than when it was OFF. The IAI Condition by Operator Workload interaction was also significant, F = 11.7, p < .005. As can be seen in Figure 14, the effect of increasing operator workload impaired the ability to maintain an optimal flight path to a significantly greater extent when IAI was OFF than when it was ON.

4.4 Airspace Violation Time

The airspace violation time data were subjected to a 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) repeated measure of ANOVA. On average, crews violated the ½ nm airspace for a significantly longer time when IAI was OFF (18.77 sec combined value) than when it was ON (3.79 sec combined value), t(7) = 2.78, p < 0.05. Regarding airspace violation time, there was not a significant difference in the effect of the IAI from one Mission Part to the next. The airspace violation time data are presented in Figure 15.

4.5 Situation Awareness

Crewmembers were asked to rate their SA for 41 individual tasks in total, including a rating of their overall SA. Each crewmember had a different number of tasks for which to provide a rating (UP = 25, UO = 24, TN = 32) as crewmembers were only asked to provide ratings for tasks that were their responsibility. Of the 41 tasks for which an SA rating was provided, only those tasks that yielded a significant main effect of IAI Condition are analyzed here. Five of the 41 tasks met this criterion.

Corrected Trajectory Score as a Function of Mission Part and IAI Condition

0

0.2

0.4

0.6

0.8

1

1 2 3

Mission Part

Tra

ject

ory

Sco

re

IAI OFF IAI ON

Figure 14: Corrected Route Trajectory Score (nm per minute) and 95% Confidence Intervals as a Function of Mission Part and IAI Condition

8 - 18 RTO-HFM-135

NATO UNCLASSIFIED

All three crewmembers were asked to provide a rating (from 1 to 7, with 1 being low SA and 7 being high SA) for their overall level of SA. Overall SA was analyzed in terms of the full 2 (IAI Condition) x 3 (Operator Workload) x 3 (Task Complexity) experimental design.

The overall SA data were subjected to a 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) repeated measure of ANOVA. Overall SA was significantly higher when IAI was ON (5.49 combined score) than when it was OFF (5.24combined score), t(7) = 2.42, p < 0.05. The IAI Condition by Operator Workload interaction approached significance, F = 3.71, p < 0.06. As depicted in Figure 16, this interaction was produced by the IAI functionality improving overall SA to a significantly greater extent for mission parts two and three than for mission part one.

Airspace Violation Time as a Function of Mission Part and IAI Condition

0

5

10

15

20

25

30

1 2 3

Mission Part

Air

spac

e V

iola

tio

n T

ime

(sec

)

IAI OFF IAI ON

Figure 15: Airspace Violation Time and 95% Confidence Intervals as a Function of Mission Part and IAI Condition

Overall SA as a Function of Mission Part and IAI Condition

4

4.5

5

5.5

6

1 2 3

Mission Part

SA

IAI OFF IAI ON

Figure 16: Overall SA and 95% Confidence Intervals as a Function of IAI Condition and Mission Part


RTO-HFM-135 8 - 19

4.6 Task Load Index (TLX)

Each crewmember was asked to rate each of the six Task Load Index (TLX) workload components (i.e., mental demand, physical demand, temporal demand, frustration level, effort, and performance) after each of the six mission parts. A rating of 0% indicates either low workload or good performance, whereas a rating of 100% indicates either high workload or poor performance. Each of these six TLX components can be analyzed in terms of the full 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) x 3 (Position Complexity: UP vs. UO vs. TN) experimental design. Of the six components, only TLX Performance is included in this results section. Five components showed a significant improvement with IAI selected ON. Only one of the six components (physical demand) did not yielded a significant main effect of IAI Condition. This is understandable because with the IAI ON and OFF, the physical demand only involved keyboard entries and low force joystick inputs. Even though the IAI functionality did not statistically improve the workload associated with physical demand, this TLX component is included in the calculation of overall TLX workload. The overall TLX workload measure is presented below.

4.6.1 TLX Performance

The TLX performance data were subjected to a 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) repeated measure ANOVA. This parameter refers to the TLX Workload associated with the need to perform. Given that a lower score for this measure of performance indicates better performance, crews reported that they performed significantly better when IAI was ON (23.4 combined rating) than when it was OFF (33.4 combined rating), t(7) = 3.40, p < 0.05. The IAI Condition by Operator Workload interaction approached significance, F = 2.93, p < .09. As can be seen in Figure 17, this interaction was produced by the IAI functionality having an increasingly larger positive impact on performance (relative to IAI OFF) for mission parts one, three, and two, respectively.

Performance as a Function of Mission Part and IAI Condition

05

1015202530354045

1 2 3

Mission Part

TL

X W

ork

load

IAI OFF IAI ON

Figure 17: Performance Ratings and 95% Confidence Intervals as a Function of

IAI Condition and Mission Part (Lower Ratings are Better)

8 - 20 RTO-HFM-135

NATO UNCLASSIFIED

4.6.2 TLX Overall Workload

The TLX overall workload data were subjected to a 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) repeated measure ANOVA. Overall workload was significantly lower when IAI was ON (31.94 combined rating) than when it was OFF (41.28 combined rating), t(7) = 3.48, p < 0.05. The IAI Condition by Operator Workload interaction was also significant, F = 4.31, p < 0.05. As can be seen in Figure 18, this interaction is produced by the IAI functionality having an increasingly larger positive impact on performance (relative to no IAI) for mission parts one, three, and two, respectively.

4.7 NAVCOM’s Assessment of Crew Performance

The NAVCOM provided a rating for each crewmember after each of the six mission parts. As such, this measure can be analyzed in terms of the full experimental design: 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) x 3 (Position Complexity: UP vs. UO vs. TN).

The NAVCOM’s assessment data were subjected to a 2 (IAI Condition: ON vs. OFF) x 3 (Operator Workload: Mission Part 1 vs. Mission Part 2 vs. Mission Part 3) repeated measure ANOVA. The NAVCOM reported that the crew performed significantly better when IAI was ON (6.08 combined score) than when it was OFF (5.42 combined score), t(7) = 3.31, p < 0.05. The IAI Condition by Operator Workload interaction was also significant, F = 6.85, p < 0.01. As can be seen in Figure 19, this interaction is produced by the IAI functionality significantly improving performance (as rated by the NAVCOM) for mission parts two and three, but not for mission part one.

4.8 Findings

The experimental results indicated that operators could manage tasks in a faster pace with less airspace violation time in part 3 (see Figures 11, 14, and 15) although the workload was higher in this part than any

Overall Workload as a Function of Mission Part and IAI Condition

0

10

20

30

40

50

60

1 2 3

Mission Part

TL

X W

ork

load

IAI OFF IAI ON

Figure 18: Overall Workload Ratings and 95% Confidence Intervals as a Function of IAI Condition and Mission Part (Lower Ratings are Better)


RTO-HFM-135 8 - 21

other part. This is because either IAI did help operators to manage some tasks or some tasks were shed by operators as there were too many to handle. Since there were less CTSs shed when IAI was ON than OFF (see Figure 12), it was possible that IAI did help operator manage some tasks. This may also be the reason why IAI had positive impact on CTS completion time and percentage of CTS shedding although it was not statistically significant. On the other hand, subjective results did reveal that operators had the highest workload in part 3 and overall SA was improved when IAI was ON (see Figures 16 and 18).

As Figures 17 and 18 illustrated, overall workload was significantly reduced and performance was improved when IAI was ON in both parts 2 and 3. Figure 16 depicted that overall SA was also improved when IAI was ON in both parts 2 and 3. When IAI helped out and/or more tasks were shed, operators could still maintain maximum SA even though they had the highest workload in part 3. Thus, it is likely that they could manage mission tasks in a faster pace with better trajectory scores and less airspace violation time (see Figures 11, 12, 13, 14, and 15). Therefore, the claim of IAI’s positive impact on high reducing workload situation and enhancing situation awareness as well as performance were supported.

In general, the results indicated that operators at all crew positions performed more effectively from both quantitative and qualitative perspectives when the IAI multi-agent system was selected ON. Some of the parameters did not show a significant improvement with IAI selected ON, however in every case the IAI ON condition was better than the IAI OFF condition providing a strong trend to the results and they were consistent in all cases. When IAI was ON, CTSs were shortened, less tasks were shed, the UAV trajectory scores were better and much less time was spent in the no-fly areas. In addition, operators’ overall SA was improved and overall workload was reduced as well. These results also validated the simulation results concluded in the first phase of this research. Operators were working in a cognitively complex situation by controlling multiple UAVs and the performance was improved through the use of an IAI.

NAVCOM's Assessment as a Function ofMission Part and IAI Condition

3

4

5

6

7

1 2 3

Mission Part

Per

form

ance

IAI OFF IAI ON

Figure 19: NAVCOM’s Assessment of Performance and 95% Confidence Intervals as a Function of IAI Condition and Mission Part

8 - 22 RTO-HFM-135

NATO UNCLASSIFIED

5.0 DISCUSSION ON IAI DESIGN

Although the IAI implemented as a prototype system in this project is only a small subset of a future more extensive suite of fully optimized UAV agent system, the experimental findings indicated that the control of a dynamic and complex system such as multiple disparate UAV control from an airborne platform can be improved through the use of a multi-agent IAI suite. More importantly, through the discussions and observations made during the conduct of the project, experience and knowledge were gained regarding the design of IAI agents, the implementation of synthetic IAI prototype environments, and the conduct of the experiments. Many thoughts were also given in terms of how to design an effective IAI system and these are discussed herein. Further to the work reported here, Hubbard, et al. (2006) hypothesized that as the level of complexity increases within these scenarios, the degree to which software agents outperform humans increases [9].

5.1 Technical Account on IAI Prototype

The experimental set-up used in this experiment was a well designed and developed environment for exploring the use of IAI intelligent agents and the provision of assistance to operators using multi-agent systems. In addition, it has been excellent for determining how to produce an IAI. It must be noted that as the project continued, the list of potential IAI functions grew based on the knowledge gained by the project team. The actual number of agents incorporated in the experiment was limited and the quality of the implementation was fair-to-good though not at the quality of a production system. Even so, the results found were very supportive of the use of IAI agents in this complex environment. It is reasonable to conclude that a more complete set of agents would result in significantly increased crew performance, certainly better than the performance obtained with the limited IAI tested during this project.

The most effective IAI agents implemented in the prototype interfaces were: route planning, route following, and inter-crew communication support agents. These three IAI components were the agents initially identified by the hierarchical goal analysis (HGA) conducted in the first phase of the project. This indicated that the methodology used for the initial agent selection was sound.

There were aspects of the experiment that limited the perceived effectiveness of the IAI. One of these was that the conventional user interface was already well designed for the control of multiple UAVs from a CP140 specifically. The second was that the crews came in for only two days and the novelty of the conventional interface would not have worn off. The crewmembers would rather work with the interface manually than give up control to the IAI. The conventional interface was designed to be effective, however the IAI ON interface still performed better. In addition, participants had been trained on the conventional interface for which they had developed work strategies and not on the IAI. With the IAI functionality selected ON, the participants may rely on the original work strategies because they were known and effective. Even so, the experimental results indicted that participants performed better when the IAI was turned ON.

5.2 IAI Design Issues

The essential purpose of this study was to develop a preliminary guideline for IAI design. Special attention was paid to this aspect during all project phases and the knowledge obtained through concept development, prototyping, and experimentation phases was documented. A subset of the knowledge gained is outlined here.


RTO-HFM-135 8 - 23

5.2.1 IAI Interface

First of all, feedback conduits are a high priority for interface design when intelligent agents are employed. This was clearly demonstrated during the graphical user interface design effort when the requirement for HGA feedback (upwards flow of information within the hierarchy of goals) was indicated as an important display item. This resulted in the IAI specific communication message window introduced in Section 2.2.

In addition, it is important that the interface allows the operator to return to the system state that was in effect before the IAI reconfigured the display or moved the camera. In other words, a “Return to Previous State” (BACK) button and a “Default Settings” button should be added to the IAI display. IAIs must be carefully designed to ensure that they are not perceived as taking over control. It is critical that the system informs the user of any changes on the interface. The IAI should either indicate for a few seconds where it is going, or indicate what has changed.

Furthermore, the design of each intelligent agent in a rapid prototype operator interface must be based on reality. The interface designer must be realistic with regards to the information available on a system data-bus. In a modelling and simulation environment, the design engineers know the ground truth, however no information may be used that cannot be obtained from a real live subsystem. This being said, during the design and development of an IAI, every attempt should be made to ensure that the intelligent agent “is aware” of the state of the world. This may include access to data fusion interim variables and associated probabilities, which would allow the IAI to produce strategies that “play the odds”. This would be a very difficult and time consuming task of the operator to complete.

It was also recognized that there is a qualitative difference in the way an IAI interface requires the operator to manage his or her UAVs. Using a conventional interface the operator provides heading and altitudes, whereas when using an IAI the operator provides goals or mission objectives. As a result, the IAI must be designed to not misinterpret intentions, for example, initiating a loiter pattern when the pilot was trying to fly to a distant refuelling location. In essence, the operators move beyond inserting parameters but rather insert or establish system objectives. The design and development team should evaluate the operator’s activities and try to move his thinking from the operating level to the strategic level.

Finally, all IAI functions which are studied during design and development, and which are incorporated into a user evaluation, should be thoroughly researched to confirm that the concept is implemented adequately for the assessment. Poor implementation will degraded the value of any IAI function, and the true acceptance or potential improvement in effectiveness will be masked.

5.2.2 Potential IAI Functionality

Due to time constraints and the scope of the project, there were only six sets of IAI functional groups (agents) implemented in the prototype interfaces. However, there are other sets of IAI candidates which are potentially good at assisting operators in effective decision-making to achieve complex mission goals, especially in the context of multiple UAV control.

First, an advantageous feature of an IAI would be to generate suggestions. For example, when a UAV is flying away for the perceived scene of action, the IAI agent may query the operator with a question similar to: “Do you want me to turn UAV 3 around?” This agent would have to recognize that a UAV was being ignored or had been forgotten.

8 - 24 RTO-HFM-135

NATO UNCLASSIFIED

Second, an IAI may build on other software in existence, which could complete part of the background data analysis. For example, the data fusion calculations based on different routes of different UAVs could lead to optimal route selection to maximize fused data. This information would be used by the IAI route planning agent.

Third, an IAI may intelligently provide information to the crew by sequentially selecting the most relevant data items for display. This could be similar to the CNN news, which continuously moves from a news story to another news story with the sequence depending on the latest breaking information and the relevance to the viewers. In this way, a UAV operator could stay attuned to the most recent video from the team of UAVs investigating contacts. This may lead to a “ticker tape” of information scrolling across the display.

Fourth, an IAI agent may produce an “intent” path for every UAV to indicate (to the operator) where each UAV will be flying. The “intent” path would be continuously reassessed and modified as required as events unfold. If a contact were to “pop-up”, the agent would re-plan routes to accommodate the new unknown. The agent could produce the new plan and offer it up for acceptance, or it could implement the new plan and modify the UAV intentions shown on the TACPLOT. Each optimal flight path may have loiter points (holding patterns) identified. A design challenge would be the level of autonomy allocated to the group of UAVs and identification of the appropriate points at which operators’ overview would, should or must occur.

Fifth, a self-preservation agent. This IAI function would maintain altitude separation between UAVs and would warn the pilot if a UAV was descending into the water. Additionally, the agent would ensure that the engine was operating and a minimal floor height maintained (with a standard holding pattern if necessary) to keep the UAV from crashing.

Sixth, another IAI agent may assess the utility of all of the UAVs. Each UAV would be assessed for utility and possible redeployment. This would identify UAVs that were flying away from the area of interest. This is a case where the agent could suggest a new route (or a loiter) and ask the operator if he or she would like to have the new route implemented, or automatically redirect the UAV to prosecute an unknown contact. Similarly, the agent could be controlled to prosecute multiple targets efficiently. For example, the agent could implement a minimum cost function to use the available resources to investigate all contacts as quickly as possible.

Finally, each IAI agent may have a different level of autonomy appropriate to the function. An example of various levels of autonomy would be the pilot accepting full route planning and route following while the sensor operator uses IAI “advise” mode rather than full turret repositioning.

6.0 CONCLUSION

Effective cognitive decision aid has been demonstrated in the context of controlling multiple UAVs by augmenting IAI agents onto operator interfaces. Positive results from both objective and subjective empirical evaluations on the efficacy of IAIs assisting UAV operators’ decision making indicated a real potential for improvement of the performance of human-machine systems, especially for those military systems carrying out cognitively complex tasks with high workload. Analyses regarding the design and implementation of most beneficial tasks for IAIs to maximize overall human-system performance were also conducted. The experience and knowledge gained from the design and implementation of IAI agents provide preliminary guidance on advanced operator interface design.


RTO-HFM-135 8 - 25

REFRENCES

[1] Miller, C. (2003). Delegation architectures: Playbooks & policy for keeping operators in charge. In Proceedings of the NATO research and technology organization special workshop on uninhabited military vehicles. Leiden, The Netherlands: June 2003.

[2] Parasuraman, R. & Miller, C (2006). Delegation interfaces for human supervision of multiple unmanned vehicles: theory, experiments, and practical applications. In Salas, E (Series Ed.) & N. J. Cooke, H. L. Pringle, H. K. Pedersen, & O. Connor (Vol. Eds.) Advances in human performance and cognitive engineering research: Vol. 7. Human factors of remotely operated vehicles (pp. 251-266). Amsterdam, The Netherlands: Elsevier.

[3] Taylor, R. M., Bonner, M. C., Dickson B., Howells, H., Miller, C., Milton, N., Pleydell-Pearce, K., Shadbolt, N., Tennison, J., & Whitecross, S. (2002). Cognitive cockpit engineering: Coupling functional state assessment, task knowledge management, and decision support for context-sensitive aiding. In M. D. McNeese & M. Vidulich (Eds.), Cognitive systems engineering in military aviation environments: Avoiding cogminutia fragmentosa! (pp. 253-312). Human Systems Information Analysis Center State-of-the-art Report 02-01. Wright-Pattern Air Force Base, OH: Human Systems Information Analysis Center.

[4] Wickens, C. D., Dixon, S. R., & Ambinder, M. S. (2006). Workload and automation reliability in unmanned air vehicles. In Salas, E (Series Ed.) & N. J. Cooke, H. L. Pringle, H. K. Pedersen, & O. Connor (Vol. Eds.) Advances in human performance and cognitive engineering research: Vol. 7. Human factors of remotely operated vehicles (pp. 209-222). Amsterdam, The Netherlands: Elsevier.

[5] Hou, M. & Kobierski, R. D. (2006). Experimental assessment of intelligent adaptive interfaces for UAV operators. (Contract Rep. CR 2006-207). Toronto, Canada: Defence Research & Development Canada-Toronto.

[6] Hou, M. & Kobierski, R. D. (2005). Performance modeling of agent-aided operator-interface interaction for the control of multiple UAVs. Proceedings of the 2005 IEEE international conference on systems, man, and cybernetics (pp. 2463-2468). Waikoloa, Hawaii, USA, 10-12 October, 2005.

[7] Hou, M. & Kobierski, R. D. (2006). Operational analysis and performance modeling for the control of multiple uninhabited aerial vehicles from an airborne platform. In Salas, E (Series Ed.) & N. J. Cooke, H. L. Pringle, H. K. Pedersen, & O. Connor (Vol. Eds.) Advances in human performance and cognitive engineering research: Vol. 7. Human factors of remotely operated vehicles (pp. 267-282). Amsterdam, The Netherlands: Elsevier.

[8] CMC Electronics Inc. (2004). Hierarchical goal analysis and performance modeling for the control of multiple UAVs/UCAVs from an airborne platform (Contract Rep. CR 2004-063). Toronto, Canada: Defence Research & Development Canada-Toronto.

[9] Hubbard, P., Fusina, G., Sachgau, M. & Culligan, I. (in production). Multiple UAV coordination: comparing human and agent performance. Proceedings of UVS Canada 2006 Conference, Montebello, Quebec, Canada, 7-10 Nov. 2006.

design and evaluation of intelligent adaptive operator

Documents