REMOTE – Desk-top VR for future command and control?R.S.Aylett1, C.Delgado1, J.H.Serna1, R.Stockdale2, H.Clarke2, M.Estebanez2,

P.Goillau2, T.Lynam2.

Centre for Virtual Environments, University of Salford, Business House, University

Road, Salford, M5 4WT, Manchester, UK 1

QinetiQ, St. Andrews Road, Malvern, Worcestershire, WR2 3PS. 2

Email: R.S.Aylett@salford.ac.uk, C.Delgado@pgr.salford.ac.uk

Abstract

The paper discusses a study in supporting collaborative military planning in which groupware, video-conferencing and a desktop Collaborative Virtual Environment (CVE) were used. It discusses the designand implementation of the CVE and the set-up and execution of the study using questionnaires andobservation. The results of the study questionnaires showed that the CVE was not seen by users as the bestof the ways offered to support collaborative planning; these results are discussed and their implication forthe design of such a CVE are assessed.

Keywords

Virtual environment, collaboration, groupware, avatar, net-VE.

1. IntroductionThis paper reports work carried out by QinetiQ and the University of Salford, both in the

UK, to compare the use of a desktop collaborative virtual environment with two other

modalities as support for a distributed military command team. The system produced,

REMOTE (Reach-back Enabling Multiple Objective Team-working Environment), was

included in an initial study evaluating potential novel command technologies for

distributed team working, and more detail can be found in the resulting QinetiQ technical

report [Clarke et al., 2003]. The customer for this work was the UK Ministry of Defence

(MoD), who are, like other national defence departments (for example the United States

Department of Defense), looking at ‘command posts of the future’. Command posts are

responsible for decision-making, the co-ordination of manoeuvres, and issuing orders to

troops, and are made up of commanders with an often substantial support staff. Such

large command posts are difficult to deploy, especially in proximity to a battlefield - not

the safest place to be - and yet there are many advantages in doing so for rapid and

efficient decision-making. As a result the idea of using technology to combine a small

mobile group with all the distributed resources they need – especially experienced people

who should not be risked by putting them close to combat - has become an attractive one.

This produces a geographically distributed team.

Much research has been carried out into distributed team working in the much broader

context of the globalisation of organisations. The issues and problems are comparatively

well known as a result, and mostly arise from the absence of spatial co-location. Co-

location has many advantages, especially for tightly coupled synchronous interaction

[Olson et al92, Covi et al 98]. A shared local context, including shared time frame and

many outside events, supports easy socialising, building trust and mutual understanding

[Bradner and Mark 02] and supporting informal interaction before and after more formal

episodes. Personal identities are known and can be taken into account – including cultural

and language differences - and peripheral interactions can be seen in passing and

interpreted. Mutual awareness is known to be central to collaborative working [Steed and

Tromp 97].

Within interactions themselves, many channels other than the overt semantic content are

available – for example gesture, gaze, posture, facial expression and prosody. It has been

estimated that more than 65% of the information exchanged during a face-to-face

interaction is expressed through these largely non-verbal mechanisms [Argyle 88], which

are mainly analogue, continuous, and support subtle distinctions and nuances. Flowing

conversation depends on many such channels for ‘back-channel’ regulation and

successful turn taking [Cassell et al 01]. Moreover, they play important roles in

establishing a common frame of reference both for concepts and objects, for example

through deictic gestures such as pointing. This in turn relies on the spatiality of such

references – people and objects are located in a common space. In multi-person meetings,

such channels can also be used to express or judge the reaction of other participants to the

current speaker. Finally, co-location minimises communication delays.

This analysis has produced a demand for ‘virtual co-location’ (VCL) – a loosely defined

concept which can be taken to mean capturing some or all of the characteristics of co-

location for participants not actually co-located, using a variety of information and

communication technologies: audio, video and computer-based. The basis for the study

reported here was the hypothesis that some of the characteristics of co-location that are

difficult to support through mechanisms such as email, electronic chat, video-

conferencing and groupware, could be supported through a collaborative virtual

environment (CVE) [Churchill and Snowden 98]. Here, a multi-user 3D graphically-

realised shared virtual space would play some of the role of shared physical space

through the sense of physical presence – the feeling of being physically located in the

virtual space – which with multiple users might then produce a feeling of physical co-

location.

2. Study methodologyThere are known methodological problems in evaluating CVEs [Steed and Tromp 97]

related to the prototypical nature of most applications and the need to deal with complex

social behaviour both inside and outside the CVE. The geographic distribution and

number of users makes conducting proper controlled experiments very difficult, while the

prototypical nature of applications usually makes it infeasibly expensive in time and other

resources to create different experimental conditions. These constraints are particularly

acute where the task involved is unpredictably structured and has an extended duration,

as in military operations (note that Steed and Tromp considered video-conferencing and

booking holidays as their applications, both much less problematic than military

operations).

In addition, studies involving military teams are inevitably hard to set up. Not only is it

inconceivable that wholly novel approaches would be tried in a live situation, even their

use in official exercises would require a degree of maturity in the technology and

acceptance by the prospective users still a long way from being achieved.

Given that this work involved an initial study, or ‘proof of concept’, participant numbers

were bound to be too small to adopt a rigorous quantitative approach, forcing a

qualitative and exploratory stance. Nevertheless, qualitative approaches are well

established in HCI work where they are seen as fundamental to user-centred design,

drawing on cognitive and social psychology and ethnography [Hughes et al 95].

Qualitative methods are also widely used in the broader field of Information Systems

[Myers and Avison 02].

The qualitative technique adopted was that of the exploratory case study. This refers to

the collection and presentation of detailed information about a particular participant or

small group, frequently including the accounts of subjects themselves [Sechrest et al 96].

The case study technique gives special attention to completeness in observation,

reconstruction, and analysis of the cases under study and deals with a situation in which

there are many more variables of interest than there are data points. This contrasts with

controlled experiments in which there are typically several orders of magnitude more data

points than variables. A case study is more concerned with establishing variables and

questions for future research than with final, generalisable conclusions, but it can also be

seen as an individual iteration in a larger process of user-centred design, with particular

relevance to use-in-context and interface and interaction issues [Shneiderman 92]. This is

particularly true where a case study can legitimately be categorized as one involving

‘typical’ users carrying out a ‘typical’ task.

Two teams of four participants made up of ex-service personnel were made available.

These were experienced personnel who had carried out such operations in their military

careers and could therefore be regarded as typical of the user group being targeted.

However these participants were only available for the week-long study in March 2003

itself, and not for prior development of the user requirements. Here, experience of past

studies in other military domains and some comment from a military link officer were the

only input. The case study approach is familiar to these users and is widely-used within

the military environment since it is a basic training method and the framework for

military exercises. One can therefore have some confidence in the typicality of the

scenarios used and their relevance to the evaluation of the technology.

As we have commented, it has often been found infeasible in evaluating CVEs to create

different experimental conditions. However it was possible to do this for the study being

discussed given that the participants were available for a week, rather than the shorter

times of some other studies [Steed and Tromp 97]. An initial view was taken that since

groupware offers the most mature technology for distributed teams, this should form the

base-line condition for the study. The package GROOVE Workspace [Groove-Networks

02] was the groupware used. Two further experimental conditions were then added, as

enhancements to, not replacements for, groupware: the addition of video-conferencing,

using CUSeeMe [CUSeeMee-Networks 01], and the addition of a CVE. It was hoped in

this way to capture differential benefits of the second and third experimental conditions.

Thus it was a fundamental requirement for the CVE that it interface with the chosen

groupware platforms and support the invocation of groupware components from within it.

Finally, it is important to remember that the target users are very unlikely to be

sophisticated users of IT and are used to coordinating activity at a distance by voice only,

using radio. They are however also used to working in a disciplined style controlled by

Standard Operating Procedures (SOPs). SOPs are detailed written instructions to be

routinely followed so as to achieve uniformity in the performance of a specific function.

They are used in many large organizations, especially where the consequences of

procedural errors can be very serious, as for example in military operations, and

experienced personnel, such as those in the current study, are familiar with the relevant

SOPs and will apply them appropriately under case-study conditions.

3. The construction of REMOTEThe experimental constraints were the starting point for the requirements of the

REMOTE software. Objects spatially located within the CVE were to be used as a means

of invoking external applications such as email, web browser, and groupware

components. A degree of fidelity to the physical environment was required – the virtual

world should look something like a command post and include objects such as a bird

table (i.e. large table for the display of maps etc) that would be the central feature in a

real command post. It was decided that a central room with other rooms leading off

would be included, with each room containing a particular piece of functionality. There

were also requirements on the user’s representation, or avatar, that will be discussed

below.

3.1 Choice of development frameworkAn initial stage of the work was to evaluate available CVE platforms. It was considered

desirable to use commercial software since this has generally higher levels of support and

fewer bugs than research platforms and is consistent with the policy of the end-user to use

COTS (Commercial Off-The-Shelf software) whenever possible. For this reason as well

as issues of availability and learning curve, the two large research platforms DIVE

[Frécon 98] and MASSIVE-3 [Greenhalgh et. al 00] were not adopted.

Three commercial packages were evaluated. These were: Adobe Atmosphere [Adobe 02],

a multi-user web virtual world package, then available in beta-release; the game engine

distributed with the game Unreal Tournament [Epic-Games 99]; and the WildTangent

software environment [Wildtangent 02], originally a games engine but now marketed as a

builder of interactive web media environments and a rapid prototyping toolkit. All of

these were capable of supporting a client-server architecture, of handling a 3D graphical

world, and of importing 3D models from widely available modelling packages such as

3ds max [DISCREET]. Atmosphere was however felt to have a rather low frame-rate and

a very limited API for adding new behaviours, only offering JavaScript. It was not clear

that it would be feasible to call other applications from within it, such as GROOVE, as

required.

The standard API for the Unreal Tournament engine was limited to its proprietary Unreal

Script, and while the Gamebots [Kaminka et al. 02] extensions make it possible to run

external code on a socket interface this seemed cumbersome. However WildTangent has

a very open API supporting Visual C, Visual Basic or Java (though it requires the

Microsoft JVM, not updated past Java 1.1.4), and is easy to integrate into web browsers,

unlike Unreal. While Unreal uses OpenGL as its underlying library, and is thus multi-

platform, WildTangent sits on DirectX and is Windows-specific, but this was not seen as

an issue for a proof-of-concept system that was intended to run under Windows. In any

case, this close relationship with DirectX makes WildTangent fast, and as it handles all of

the graphics, there is little speed penalty in using Java as a framework around it with the

accompanying benefits of a modern web-oriented language. Thus WildTangent was

chosen as the development platform and Java was used to access its facilities.

3.2 Embodiment and identityIn considering what user representation to support within REMOTE, certain basic

considerations were important. User representations – avatars1 – are added to a CVE to

meet six generic requirements, as proposed byPandzic et al 97: (i) Perception allows a

user to register the presence of others, and (ii) localization to see where in the

environment they are. (iii)Identification is then the ability to know who they are. These

three capabilities are basic but can be met by fairly crude graphical representations.

A further three capabilities require more sophisticated representations. (iv)Visualization

of interest focus usually requires the representation of gaze direction, most obviously

through a user representation with eyes, whether realistic, as in a humanoid figure for

1 We use the term avatar here in its original VR sense [Stephenson 93] to mean any representation of the user in thegraphical world, not in the more recent sense of any humanoid virtual agent, user-driven or not

example or iconic, such as a personal emblem or object. This is also achievable using a

sensor beam as in virtual robots. (v) Visualization of actions can only occur if the avatar

has the ability to carry out actions, using some form of end-effector, in robot terminology.

This could be a robot gripper, or arms and legs, but again, could also be represented very

simply by extending a line to a manipulated object, or through a 3D cursor or virtual hand

grasping the object. (vi) Avatar Communication poses the heaviest requirement - in real

life, as already argued above, it is supported by gesture, posture, facial expression and

vocal characteristics. Creating an expressive avatar requires a relatively sophisticated

representation, which may then create a problem for the user who has to manipulate this

expressive repertoire while also carrying out whatever collaborative task is at issue.

Perception, localisation and identification of course include self-knowledge too, and this

depends on whether a first-person or third-person view is adopted. In the former case, the

user does not see their own avatar since their position is as if they were looking through

the avatar’s eyes; in the latter case, the user is positioned a little above and behind the

avatar. Atmosphere and Unreal Tournament both take a first-person view (with the

weapon being carried visible in Unreal) but a third-person view was selected in

REMOTE. In the real world we always have some awareness of what we look like and

where our body is in relation to objects around us, and this effect of the third-person view

was felt to outweigh the inconvenience of having the avatar body partially block the

user’s field of vision.

A humanoid form for an avatar is not always essential, but in terms of expressiveness it

has many advantages [Benford et al 97]. It seemed especially relevant to REMOTE’s

target users given their lack of familiarity with graphical environments. It also provided

an intuitive method for displaying the service, rank and role of a particular user,

significant identifiers in military organizations and especially in control rooms where

commanders of all services may collaborate along with civilian or government

specialists.

A natural way of identifying individuals would be to texture their face onto the face of

their avatar, technically quite straightforward. The conditions of the study made this

impossible, as the participants were unknown until shortly before it took place; instead

the name, role and rank of the user were attached above the head of the avatar, using a

billboard so that it always faced towards a viewing user, the data being gathered at logon.

However this was removed from the user’s own avatar (they know who they are) as it

blocked too much of the field of view. Uniform was used to indicate service membership

– camouflage for army, white shirt and dark blue trousers for navy, and light blue shirt

and blue trousers for air force. Both male and female avatars were provided, the gender

user-selected at logon, and figures of somewhat heroic physical proportions were

produced (see Figure 1). Had timescales allowed it, different character designs would

have been developed and evaluated with end-users: as it was, only feedback from the

military liaison officer was available and his response to this design was positive.

Figure 1 Avatars in REMOTE

As already suggested, there is a trade-off between the communicative advantage of giving

avatars more expressive behaviour and the disadvantage of producing a cognitive

overload for users in trying to invoke such behaviour appropriately. A small set of

animations was therefore provided as seen in Figure 2. It was seen as particularly

important to indicate when the avatar was inactive because the user driving it was absent

– the ‘chill-out’ animation.

The rotate and walk animations were invoked when the user navigated their avatar within

the environment. However it was also considered important to a feeling of physical

presence and realism that avatars could not walk through walls or furniture, so that

obstacle avoidance was required. The method of separating axes was used to determine

whether or not two bounding boxes intersected. Using the data for the centre, axes and

extents of the avatar’s box and for the bounding box of the object to be tested, classical

algorithms to test if two boxes have an intersection as described in Eberly 01 can be used.

S t i l l

C h i l l - o u t ( n o t P r e s e n t)

W a l k

R o t a t e

R u n

S a l u t e

l e a n

s t a r t

If they do collide then the velocity of the avatar is reduced to 0 and the position is

updated accordingly.

Latency is always an issue in a

CVE, and if the position of an

avatar is updated only on

coordinates from the server,

latency effects can produce a

very jerky animation. Dead-

reckoning can be used to

improve this situation by

making a local prediction in the client about where an avatar is likely to move, moving

towards that point and updating with the incoming coordinates from the server. Thus

dead reckoning consists of two parts prediction and convergence [Singhal and Zyda 99].

Prediction is the way the entity's current state is computed based on previously received

update packets while convergence defines how the object’s position and trajectory is

corrected as in the following equation:

Pred.pos after t secs = pos + velocity x t

The difference between the real position and the predicted position is next computed and

a new path is calculated, which is used in the convergence algorithm. Finally a point in

the future is selected on the new predicted path, and the navigation is updated

accordingly. Several techniques have been proposed [Singhal and Zydah 1999], although

a linear convergence (using velocity but not acceleration) was selected for REMOTE,

Figure 2 State Diagram of animations.

because latency was not thought to be a big problem in the setup of this study given that

it ran on a LAN in two neighbouring rooms.

3.3 Interacting with other applications

Object Room Action

Board Main room Launch Newsgroup

Table with map Bird table room Groove sketchpad tool

Cabinets Files room Groove files tool

Board Pictures Room Groove Pictures tool

Presentation dais with stairs Meeting room Nothing

Flying doc Documents room Groove notepad tool

Other Avatars All Netmeeting call to Avatar's IP (was

not used in actual test)

Table 1 Objects and locations in REMOTEAs discussed above, the ability of the CVE to interact with other applications was a

fundamental requirement. Applications were associated with particular objects in the

environment, mostly located in specific rooms, each labelled with the function. Table 1

Figure 3:REMOTE CVEand launchedapplications.

shows the objects, associated applications, and the room in which they were located.

A bounding box was implemented around the launch objects and the associated

application for a user launched when their avatar intersected the bounding box. To make

this visible to the user, the bounding box was rendered as partially opaque on intersection

– as in the top left component of Figure 3, in which the Board object in the Pictures room

has been intersected, bringing up the GROOVE Picture seen bottom right in Figure 3.

When the avatar leaves the bounding box the application is closed, and a delay factor was

incorporated to prevent rapid zigzags between opening and closing. Interactions with

another user were initiated by approaching that user’s avatar, and such interactions were

limited to one-to-one. The main purpose here was to invoke Netalk to allow voice

interaction between two users. The bottom-left component of Figure 3 contains the

logging file listing specified events in the VLE, though the text is not readable in this

diagram, and the top-right component contains the login facility.

3.4 Overall architectureFigure 4 shows the client-server architecture of the overall system. Client-side servers

handle the invocation of GROOVE and other applications, a login application records

significant actions of the avatar and the login functionality handles the identification of

users and their attachment to an avatar.

The architecture is divided into two logical components: the client and the server side.

The server side handles the data received from the clients, that is positions and actions of

the avatar and it replicates the data to the clients so that the all clients have the data

regarding the other users' avatars. To handle communication between the server and the

clients, the Observer-Subject Pattern was used, updating information when a message is

received by the listener (as seen in figure 4). This runs in a separate thread.

On the client side, (Figure 4) several subcomponents were developed to handle the

required tasks. Object Orientation and Design Patterns [Gamma et al, 1996] are heavily

used in the architecture offering support for modularity and the use of existing abstract

solutions to common problems like the use of controllers (the singleton pattern was used

to have just one instance of a class of controllers in the code). These controllers are used

to handle the animation of the avatars, the network communications and the data relating

to all the objects displayed in the virtual environment.

Also important was the use of the command pattern to link an object in the virtual

environment to an action, for example to open a GROOVE subcomponent. GROOVE is

itself a set of subcomponents, and as Table 1 shows, four of these – sketchpad, files,

notepad and picture – were invoked by an avatar approach to the relevant object. This

was handled by the GROOVE Server: the object has an associated message, which is sent

to the server, creating a link between object and GROOVE subcomponent. A similar

mechanism was used to open other Windows applications using the application server,

Figure 4 Overall architecture diagram.

and an editable XML file was used to specify the repertoire of connections making the

system easily extensible.

Another external program is a Logger Server, used to store copies of messages so that a

history of the interactions between avatars and objects can be maintained. This was also

configurable via XML. The login is used to enter the profile of the user and allows a user

to choose their sex (male, female), their service, their role and their rank. The selected

profile is used to generate an avatar in the appropriate uniform and the label above the

avatar's head.

4. Study and resultsAs already mentioned, the study involving the use of REMOTE ran in March 2003 and

involved two teams of four ex-military personnel, each forming a HQ. A team contained

a Chief of Staff (COS), an Intelligence coordinator (J2), Operations controller (J3) and

Logistics controller (J4). These are standard and well-understood HQ roles. Each team

was located in a separate room but shared the same LAN. No attempt was made to

simulate latency, bandwidth constraints or other WAN conditions.

4.1 Use of scenariosThe scenarios were all set within the context of a UN peace-keeping operation in the

fictitious country of New Albion, an island archipelago in the North Atlantic

approximately 600Km to the west of UK. New Albion comprises three ethnically divided

provinces: Wessex (Angle), Siluria (Celtic) and Devonia (Celtic). After 10 years of bitter

civil war the UN is invited to intervene to monitor a lasting ceasefire, however UN

intervention fails through lack of a clear mandate and other factors. A new ceasefire is

brokered and NATO agrees to intervene with New Albion consent and a revised UN

mandate. The US, UK and France each agree to send Divisional sized forces (NAFOR).

Despite the ‘ceasefire’ there is an operationally urgent need to get new forces on the

ground as quickly as possible. The UK contributes a framework Multi-National

Divisional HQ, some Divisional troops, a Mechanised Brigade and an embarked

Commando Brigade.

The background information remained the same throughout all three conditions,

minimising the additional read-in time each day and allowing more time for planning.

Each of the three experimental conditions described above in section 2 was introduced by

a new task set within the overall scenario. For condition 1, this was to organise the relief

of the commando brigade by the incoming mechanical brigade; for condition 2 to deploy

forces to secure a town in which ‘ethnic cleansing’ was threatening; for condition 3 to

evacuate threatened settlers from the town back to their ethnic home area and protect

those staying on. The output from each scenario was a plan briefing in which a remotely

situated representative Commander was presented with potential courses of action

(COA). Scenarios were such that they deliberately forced collaboration between HQs -

for example, sources of information were placed at different levels within each HQ. A

control cell ran the simulation, generating military and political events.

The maintenance of Situational Awareness (SA) is usually ranked among the most

important tasks for all command post team members [Taylor 1990] and experienced

personnel such as those used in the study perceive it as a key task. A widely (though not

universally) accepted definition is that of [Endsley 95] in which SA is the perception of

the elements in the environment within a volume of time and space, the comprehension of

their meaning, and the projection of their status in the near future: informally ‘what is

going on’. Military planning in command posts is naturally hierarchical (for example

strategic, tactical or operational), and in the scenario used, planning invariably

commenced with one HQ having better higher-level SA while the other would have better

lower-level SA. This was achieved by having one HQ ‘on the ground’ operating at the

tactical level and the other working at the operational level out of the area.

The study used a within groups (related, same-subject) design [Clarke et al 03] so that all

subjects performed all conditions: GROOVE alone, GROOVE + CUSeeMe and the

GROOVE + REMOTE. The constraints on the availability of participants, and the

resources available, made this the most feasible approach. There are known difficulties

with this design: for example, there can be a learning effect from one condition to the

next and a potential order effect where performance either improves as a result of practice

and familiarity or degrades because of fatigue or boredom. Changing the scenario and the

role of the team members between the conditions can counterbalance some of these

effects. However, this needs to be taken into account when analysing the results and is

discussed in section 4.4 below.

In the five-day programme, each 4-person team spent the first day separately carrying out

non-technology-based team-building exercises, to avoid an increasing team familiarity

and cohesion among the co-located participants impacting the study. During the second

day, familiarisation with GROOVE was carried out. Then one condition was run on each

day, with the last part of days 3 and 4 spent on training in CUSeeMe and REMOTE

respectively. The last part of day 5 was used to complete the questionnaires discussed

below and for a debrief.

4.2 Data collection and analysisData was collected by observations made during each experimental condition and through

questionnaires completed after each experimental condition, see Appendix 1. Two

observers were present in each team room and used an observational ‘crib sheet’ as a

guide – see Appendix 3. Observations covered: the human factors issues associated with

working in a distributed command environment; how the tools supported the VCL

concept though team and task work; the effects of the different tools on local and

distributed team interactions; the effect of the tools on the military task; difficulties in

using the tools. A set of categories was developed to allow observations to be grouped

and this can be seen in Appendix 4. The debrief was carried out using the prompt sheet in

Appendix 2.

Three questionnaires were used, containing a 7 point Likert rating scale and completed

electronically. The three questionnaires covered: aspects of distributed and collaborative

team working effectiveness; the degree to which the collaborative technologies meet

VCL (Virtual Co-Location) requirements; and aspects of the collaborative technologies

including general comments / strengths / weaknesses / missing features, and tool design

aspects and the effect of the collaborative technologies on co-located team working.

Subjects were invited to add any comment they wished after any question. The debrief

followed the questionnaires and was structured around five ‘open’ questions and

associated prompts as seen in Appendix 2.

4.3 Results summaryThe averaged responses to the VCL requirements questionnaire, which was composed of

41 statements each with a 7-point Disagree-Agree response scale, are shown as graphical

profiles in Figure 5. There was considerable variation in responses across individual

questions, of which all but two – questions 2 and 9 - were posed positively (i.e. higher

score = better). Overall, the GROOVE + CUSeeMe condition achieved higher

‘agreement’ ratings than did the GROOVE alone condition, which in turn was rated

higher than the GROOVE + REMOTE condition. Overall means across all conditions

and participants were 4.87 (GROOVE+CUSeeMe), 4.12 (GROOVE alone) and 3.38

(GROOVE+REMOTE). Separating these profiles by team showed that Team 1 rated

GROOVE alone and GROOVE+CUSeeMe similarly to team 2. However,

GROOVE+REMOTE achieved higher agreement ratings from Team 2 (overall mean

3.86) than Team 1 (overall mean 2.9).

This difference appeared to relate to the very different ways of working of the two teams:

observation logs showed that Team 2 spent far more time in own-team meetings to

discuss plan options, details of how they were going to go about the task, the information

that they had available to them, initial ideas, and the tasks each of the team members

would carry out. They would most often turn away from their workstations to hold such

meetings and as a result message alerts and calls from team 1 would be missed. Team 2

members also stood up to leave their workstations and temporarily gather around other

members’ workstations to discuss information far more than team 1 members. A

consequence of this was that Team 1 members were frequently unable to contact Team 2

members and expressed a great deal of frustration. REMOTE had no facility to signal

when such own-team meetings were taking place and thus contributed to rather than

relieving Team 1’s frustration.

REMOTE’s best scores related to Q11 (‘The software enabled me to rapidly exchange

key information with another individual on a one-to-one basis.”) and to Q21 (“The

software enabled all members of the team to undertake their roles and responsibilities”)

strongly suggesting that the overall critical response was not due to functional

incapability or fragile implementation, but rather to the basic design decisions made. Its

worst scores relate to Q18 (“The software effectively supported social interaction (e.g.

chatting or interacting separate to the work task) with another individual on a one-to-one

basis.”) and Q19 (“The software effectively supported social interaction with several

other team members at the same time.”), reflecting the constraint of one-to-one

communication only.

In terms of meeting distributed team members as easily as co-located members,

REMOTE was overall considered better at supporting this than GROOVE, but was

considered worse than CUSeeMe. This was due to an exceptionally high rating by Team

2, which judged REMOTE superior to both GROOVE and CUSeeMe in this regard.

Team 2 rated REMOTE on a par with CUSeeMe, however, in its ability to support a

sense of shared meeting space.

The ranking between the software of the three conditions shown in these subjective scales

is reflected in the participants’ written questionnaire comments, and observers’

observations. The two other questionnaires alternated positive and negative questions, so

that the result profiles are less easy to visualise than the one shown, but they also gave a

compatible ranking of the study conditions.

We return here to the issue raised in 4.1 above: whether there was a learning effect from

one condition to the next and a potential order effect either improving performance as a

result of practise and familiarity or degrading it because of fatigue or boredom which

might itself explain the ranking of the three conditions. First we would restate that the

first day included a scenario based team task without any of the software under

examination in order to minimise the effect of improving social cohesion among subjects

who were new to each other. Secondly, GROOVE was in use for all three days of the

experimental condition as well as in training on the second day. A positive learning effect

for it as between condition 1 and condition 2 was noted in observation. However had this

effect been dominant, then one would not expect the second experimental condition to be

ranked higher than the third one, as it was. If a boredom factor for GROOVE was

involved, one would have expected the observation process to detect it, and it did not.

The area in which boredom and frustration may have impacted the study lay in the

difference between Team 1 and Team 2’s work styles, and the frustration suffered by

Team 1 as a result discussed above. This may have been a further factor in the more

critical assessment of the third experimental condition by team 1. However both teams

ranked the three experimental conditions the same, suggesting that the order effect, if

present, was not decisive. In general, the observation regime acted as a control on both

factors since it produced a detailed picture of action and interaction.

4.4 User -identified problemsThe overall questionnaire results above show that users were critical of REMOTE

compared to GROOVE and CUSeeMe. The underlying problems prompting these results

are brought out in user comments which were collected by the observers, made during the

debrief session, and included in the questionnaires. Table 2 lists the most significant

problems, illustrated with specific comments, which are naturally there taken out of the

context in which they occur. Nevertheless, the typicality of the users and of the scenarios

asserted in section 2, gives us some confidence that our identification of the generic

problems is correct. Discussion follows in the next section.

Positive comment related to creating a feeling of presence and good awareness of other

team members and what they are doing, for example: “The main strength is that it creates

a feeling of the team being present especially the distributed team.”; “The avatars provide

good awareness of other team members and what they are doing”; “ CVE provides a good

sense of virtual presence”.

FIGURE 5: VCL requirements results

0

1

2

3

4

5

6

7

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41

VCL requirements

Dis

ag

ree

- A

gre

e 1

-7

T2 Groove T2 CUseeMe T2 REMOTE

Problem Comment

Navigation slow and

laborious

J2: "Where do I find the notepad?", COS: "In the documents room", J3: "see you there in 10

minutes!"

"I just can't see the logic of it…it's just an arcade game!". Moving people around the

environment is laborious. I can't see the point of moving mannequins around a set of

corridors".

COS: "OK, Let's all go to the bird table", J3: "Could take a while".

Difficulty in finding

colleague’s avatars

"Where is he? He was in the files room, he was there a minute ago". (At the central table)

"OK, he's landed".

Only one app at a time

can be used

"Can I put a map in there/ I want to be able to drag and drop. Need to import the map, but

can't import the map via sketchpad. It's hopeless. If I can't load a map into sketchpad, can

only start from scratch."

"If I want to work on a file, I have to come out of notepad".

”Difficult to know whether individuals in different teams are looking at the same map. Quite

difficult to confirm, as you can't look at the map and chat at the same time”

“The REMOTE environment forces you to only take in one piece of information at a time.

Splits your attention.”

Avatar jams by app

launchers – see Fig 6

"Queuing up at filing cabinets with 4 big guys in the way".

Wrong information

access metaphor

"To look at a file, we need to leave the meeting room, bump into walls and go to the files

room…. I think that's ridiculous".

"You're constantly thinking where is everything… and how do I get there?"

"You shouldn't have to go into another room to get something"

"I just think the analogy is wrong with rooms etc. You put together your folder and carry it

around with you".

Only 1-1 voice

interaction

"This is even worse, we're just one-to-one chatting now". "I'm going to leave this now,

because there's no point"

Table 2. User-identified problems

5. Discussion and Further WorkThe results of the study show that the REMOTE CVE was not seen as the best way of

supporting collaborative team working by its users. However one must take into account

here the inherent difficulties in supporting distributed teams with technology. The

following comment is instructive, arising from a study into the use of non-CVE

collaborative tools to support distributed design teams in manufacturing engineering:

“Findings from this study suggest that collaborative tools must be clearly superior to

existing practices to merit the effort of deployment, adoption and subsequent use, since

the burden of learning and mastering a new tool in a demanding and fast-paced

engineering design environment may not outweigh the perceived benefits.” [Wierba et al

01 p1].

In a similar collaborative design environment, albeit with physically co-located teams, the

technology that was most clearly valued was a printing wallboard that saved participants

the time previously spent copying informal diagrams to take away [Olson and Olson 00].

Here, ‘just one thing’ was added to existing and familiar technologies. Work in the field

suggests that involvement of users in the design of collaborative environments – difficult

to set up in military domains – plays a very important role. For example, a successful

‘collaboratory’ of space physicists [Olson and Olson 00] has involved 10 major redesigns

over 7 years, all involving the users. In this collaboratory, scientists organise themselves

into virtual rooms of objects and remote partners to run experiments using real-time data

from geographically distributed instruments, suggesting that the basic metaphor of the

virtual room may play a positive role.

Figure 6 Avatars queuing to access the File resource.

The key question is whether a CVE is inherently incapable of delivering clearly superior

results both to existing practice and other technological solutions such as the two first

experimental conditions, or whether a different set of design decisions, together with

some maturing of underlying technologies, might produce a more successful CVE. This

question is related to an ongoing ‘space-versus-place’ debate within the CSCW

community as summarised in [Benford et al 2001].

The arguments for the ‘space’ metaphor already outlined suggest that it is independent

movement within a shared coordinate system, combined with the representation of others'

positions through avatars that allows a CVE to support social interaction. Those

supporting the ‘place’ argument suggest that other important aspects of an environment

beyond the provision of a shared coordinate system are responsible for supporting social

interaction. We are more convinced by the potential benefits of a shared space, and here

put forward lessons learned from the study on the design of such a space for distributed

military command teams.

REMOTE applied a relatively naturalistic design to its virtual world, while interfacing it

to applications such as groupware with a desktop metaphor. These two metaphors did not

appear to marry together well for the users, and the conflict between them was sharpened

by locating each application launcher in a different virtual room. Unlike the collaboratory

cited above, in which an experiment is a specific activity and thus fits conceptually into a

separate room, military planning requires a strong integration of different information

sources from different locations.

Thus the bird table plays a very central role in real command posts and this was not

successfully replicated in REMOTE. One user commented: “The key to digitisation is a

version of the electronic bird table. Showing the same picture to all team members, and

can plan concurrently”. REMOTE applied a naturalistic metaphor for physical space, this

was not true of its information metaphor: “Compared to the analogue system, you have a

map and notepad which you carry with you anywhere. So you shouldn't have to go to

other rooms - not realistic". Allowing avatars to carry information sources would also

make it natural to have many open simultaneously and support transfer between them –

the restriction of having one tool only open at a time was felt acutely, especially .for

maps, often used with other documents, and a natural focus for voice interaction. It would

also have avoided queues around central resources such as in the Files Room (Figure 6).

The consequence of the decision to distribute information sources was a high requirement

for navigation. The effort this produced for users underlines a basic difference between

real and virtual spaces. In the real world, most spatial navigation in office environments

is carried out at a sub-cognitive level – we walk and think, or talk, at the same time. This

is not true of navigating an avatar in a desktop CVE and it is hard to conceive of a

navigational mechanism that would be as low in effort. But this does not mean that a 3D

space is in itself a bad thing, merely that one should not try to ‘walk’ avatars about a large

space with separate rooms as in REMOTE.

A further consequence of the design was that users found it hard to locate the avatars of

users with whom they wanted to interact. A ‘plan view’ was suggested as a result, and

this together with some colour-coding would support easier location. A single-room

design would fundamentally reduce the problem, but might then meet over-crowding

problems in a command post with many more participants. Centering interaction on

avatars rather than information sources might produce a single-room design with ‘team

areas’ or stations for particular roles to interact – an intelligence or a logistics desk for

example.

The key issue for these users is visualising the state of the task; one user commented “the

environment doesn't change, it's the task that changes”. This is not entirely true of course

– the changing element in the environment is however the avatars themselves, so that

their identity and activity is what the environment really ought to communicate. However

supporting the visualisation of maps, the key information source for military planning,

would be a better use of 3D for these users: “Really good (virtual) 3D mapping would be

excellent. Especially if you are able to work with and annotate the maps.”

The identification mechanism adopted of labels over avatars’ heads was not completely

successful. Such legends may be hard to read – as can be seen in Figures 1, 3 and 6 - and

at a sensible size take up too much space. Texturing avatar faces with those of the users

themselves ought to be tried –not possible in this study because the identity of the

individuals participating was known only a short time before the study took place.

However individualised faces along with further clothing cues might have been

attempted. As with navigation, it is hard to think of a currently viable mechanism to allow

a user to invoke complex expressive features in an avatar without imposing a high degree

of effort. The positive response of the users to web cam data suggests that compositing a

video head-and shoulders onto the avatar’s face is worth considering in spite of the

technical difficulties, at least when voice interaction between users takes place.

User responses underlined the importance of voice interaction in collaboration, and the

importance of multi-voice conferencing. In spite of the use of a LAN there were still

problems with audio quality in the experiment, and this has been confirmed by a number

of earlier studies [Tang et al 94; Olson and Teasley 96; Finholt and Olson 97; Covi et al

98]. The voice facilities of GROOVE were criticized, and though NetMeeting was seen

as better in quality, participants would have preferred to use the phone. Voice-over-IP is

still less reliable and of a much lower quality than telephony. Attempts to integrate

NetMeeting into REMOTE along with the other applications were not successful, so that

it had in fact to be run in parallel outside the environment instead of being triggered when

two avatars met. Even then, the restriction to one-to-one conversation was far too

constricting.

6. ConclusionsRunning studies such as this one and learning from the results seems fundamental to the

development of CVEs into a successful and useful technology. Collaborative activity is a

particularly difficult area to support with technology, and this is all the more so for

tightly-coupled tasks with a high degree of inter-dependence such as in military planning

[Olson and Olson 00]. A deep understanding of the user requirements and how these can

be supported with the technology as it now is cannot be achieved without such

experimental work. In this case, a desktop VE was poorly received, but as we have tried

to indicate in this final section, this seems more a case of iterating the design concepts

than abandoning the approach. There is still a need to assess more explicitly the impact of

these technologies on task performance and conduct to determine whether they provide

the ability to enable the task to be done more quickly and more efficiently.

ReferencesAdobe (2002). Adobe Atmosphere, http://www.adobe.com/products/atmosphere/main.html.

Argyle, M. (1988) Bodily Communication , New York: Methuen and Co., 1988.

Benford, S. D., Bowers, J. M., Fahlen, L. E., Greenhalgh, C. M., and Snowdon, D. N. (1997) Embodiments,

Avatars, Clones and Agents for Multi-user, Multi-sensory Virtual Worlds, Multimedia Systems, 5, 2, 1997,

93-104.

Benford, S., Greenhalgh, C., Rodden, T. and Pycock, J. (2001) Collaborative Virtual Environments,

Communications of the ACM, Volume 44 , Issue 7 pp79 – 85, July 2001

Bradner, E. and Mark, G. (2002). Why Distance Matters: Effects on Cooperation, Persuasion and

Deception. In proceedings of Computer Supported Cooperative Work (CSCW 2002). November 16-20,

New Orleans, Louisiana

Cassell, J., Nakano, Y., Bickmore, T., Sidner, C., and Rich, C. (2001). “Non-Verbal Cues for Discourse

Structure.” Proceedings of the 41st Annual Meeting of the Association of Computational Linguistics, pp.

106-115. July 17-19, Toulouse, France

Churchill, E and Snowdon, D (1998) Collaborative Virtual Environments: an introductory review of issues

and systems, in Virtual Reality: Research, Development and Application, 1998

Clarke, H; Goillau, P and Stockdale, R. (2003) Distributed Teamworking Experiment, QinetiQ May 2003,

Malvern UK

Covi, Lisa M., Olson, Judith S., Rocco, Elena, Miller, William J., and Allie, Paul. (1998) A Room of Your

Own: What do we learn about support of teamwork from assessing teams in dedicated project rooms. In

Cooperative Buildings: Integrating Information, Organization and Architecture: First International

Workshop, Co'Build '98. Darmstadt, Germany.

CUSeeMe-Networks (2001). CUSeeMe, http://ic2.cuseeme.com/.

DISCREET : http://www.discreet.com/support/max/

Eberly, D. H. (2001). 3D Game Engine Design, San Francisco: Morgan Kauffman Publishers.

Endsley. M.R. (1995) Toward a theory of situation awareness in dynamic systems. Human Factors ,37 (1),

32-64.

Epic-Games (1999). Unreal Tournament, Infogrames.

Frécon, E. and Stenius, M. (1998). DIVE: A scaleable network architecture for distributed virtual

environments, Distributed Systems Engineering Journal (DSEJ), 5, pp 91-100, Special Issue on Distributed

Virtual Environments.

Finholt, T. A., and Olson, G. M. (1997) From laboratories to collaboratories: A new organizational form

for scientific collaboration. Psychological Science. 8(1), 28-35

Gamma, E; Helm, R., Johnson, R and Vlissides, J. (1996) Design Patterns, Addison-Wesley Longman,

Reading Massachusetts.

Greenhalgh, C., Pubrick, J., and Snowdon, D. (2000). Inside MASSIVE-3: Flexible support for data

consistency and world structuring. In Proceedings of the 3rd International Conference on Collaborative

Virtual Environments, (CVE ‘2000), San Francisco, California, USA, ACM Press.

Groove-Networks (2002). Groove Workspace, http://www.groove.net.

Hughes,J; King,V; Rodden, T and Anderson,H. (1995) The Role of Ethnography in Interactive Systems

Design, ACM interactions , v.2 n.2, p.56-65, April 1995

Kaminka, G. A.; Veloso, M.; Schaffer, S.; Sollitto, C.; Adobbati, R.; Marshal, Andrew N.; Scholer,

Andrew, S.; and Tejada, S. (2002). GameBots: the ever-challenging multi-agent research test-bed, In

Communications of the ACM, January issue.

Myers, M.D. and Avison, D.E. (eds.). (2002) Qualitative Research in Information Systems: A Reader, Sage

Publications, London, 2002.

Olson, J. S., and Teasley, S. (1996) Groupware in the wild: Lessons learned from a year of virtual

collocation. Proceedings of the Conference on Computer Supported Cooperative Work. New York: ACM.

pp-419-427

Olson, G.M., and Olson, J.S. (2000). Distance matters. Human Computer Interaction, 15, 139-179.

Pandzic, I; Lee, E; Magnenat Thalmann, N; Capin, T and Thalmann, D. (1997) A flexible architecture for

Virtual Humans in Networked Collaborative Virtual Environments Computer Graphics Forum Vol 16,

Issue 3 Conference Issue (September 1997)

Sechrest, L., Stewart, M., Stickle, T., and Sidani, S. (1996). Effective and Persuasive Case Studies.

Cambridge, MA: Human Services Research Institute.

Shneiderman, B. (1992). Designing the user interface: Strategies for effective human-computer interaction,

2nd edn, Reading, MA: Addison-Wesley

Singhal, S., and Zyda, M. (1999) Networked Virtual Environments, New York:ACM Press, Siggraph

Series.

Steed, A and Tromp, J. (1997) "Experiences with the Evaluation of CVE Applications", in Proceedings of

Collaborative Virtual Environments 1998, 17-19th June 1998, Manchester, UK

Stephenson, N.(1993) Snowcrash. Spectra Books , 1993.

Tang, J. C., Isaacs, E. A., and Rua, M. (1994) Supporting distributed groups with a Montage of lightweight

interactions. Proceeding of the ACM Conference on Computer Supported Cooperative Work (CSCW94),

Pp. 23-34

Taylor, R. M. (1990) Situational awareness rating technique (SART): The development of a tool for

aircrew systems design. In Proceedings of the Situational Awareness in Aerospace Operations, AGARD

CP-478, Paper 3.

Wierba, E.E., Finholt, T.A., and Steves, M. (2001). "What have you done for me lately?" A case study of

barriers to collaborative tool adoption in a manufacturing engineering setting (PDF file). At:

http://www.crew.umich.edu/technical_reports.htm

Wildtangent (2002). Wildtangent Web Driver, http://www.wildtangent.com

Appendix 1 – The Questionnaires

A1.1 the team-working questionnaireQUESTIONNUMBER Questions

1 In the co-located team, it was possible for team members to have a similarunderstanding of the situation.

2 In the distributed team, it was possible for team members to have a similarunderstanding of the situation.

3 When the team was co-located, technology hindered the team members’understanding of the situation.

4 When the team was physically located together, it was possible to have anunderstanding of the overarching team goal.

5 In the distributed team, it was difficult to maintain an understanding of wherethe team fitted into the bigger picture.

6 It was easier to have an awareness of other team member’s roles andresponsibilities in the co-located, rather than the distributed, team

7 In the co-located team, it was easy to monitor and assess the progress of fellowteam members.

8 In the co-located team, it was easy to ask for guidance from the Chief of Staff.9 In the co-located team, it was difficult to notice and prevent team problems.10 It was difficult to notice and prevent team problems, when working in a

distributed team.11 It was more difficult to resolve conflicts when team members were distributed.12 It was difficult to know when to provide constructive feedback to team

members who were not in the same room.13 It was easy to identify when co-located team members required assistance or

support.14 It was easy to ask for support when the team was co-located.15 Feedback was more discrete and less formal when given face to face.16 In the distributed team, it was difficult to know when other team members were

having task-focused problems.17 It was easier to develop trust between members in the same room than team

members that were distributed.18 When separated from the rest of the team, it was possible to experience feelings

of isolation.

of isolation.19 It was easy to develop and maintain interpersonal relationships between co-

located members of the team.20 The technology hindered the development and maintenance of interpersonal

relationships.21 Unfamiliarity within the distributed team, prevented team members from

asking for support.22 It was easy to build and maintain a good team atmosphere in the co-located

team.23 In the co-located team there was a high level of satisfaction.24 The technology hindered the overall level of satisfaction in the co-located team.25 In the distributed team there was a high level of satisfaction.26 The technology hindered the overall level of satisfaction in the distributed

team.27 In the co-located team there was a high level of motivation.28 The technology hindered the overall level of motivation in the co-located team.29 In the distributed team there was a high level of motivation.30 The technology hindered the overall level of motivation in the distributed team.31 In the co-located team, there was high level of cohesion.32 The technology hindered the development and maintenance of cohesion in the

co-located team.33 In the distributed team, there was a high level of cohesion.

34 The technology hindered the development and maintenance of cohesion in thedistributed team.

35 It was difficult to interact and communicate with co-located team members.36 When team members were not co-located, additional co-ordination and

communication demands arose.37 It was difficult to interact simultaneously with co-located and distributed team

members.38 It was difficult to conduct brainstorming sessions with co-located team

members whilst maintaining awareness of the activities of distributed teammembers.

39 It was difficult to maintain eye contact with co-located team members.40 In the co-located team, it was difficult to manage team members’ tasks and co-

ordinate a unified effort.41 When a message was not communicated verbally, face-to-face, it was difficult

to understand whether a team member had understood the message.42 Requesting information was easier when face-to-face.43 In the distributed team, it was difficult to know how and when to update others

with information.44 Lack of face to face communication increased the number of

misunderstandings that occurred.45 Communicating over electronic media was not as effective as face to face

communication.46 The physical layout of the work environment inhibited non-verbal

communication between co-located team members.

A1.2 The Requirements Questionnaire

QUESTIONNUMBER Questions

1 The software did not constrain team working between members of the co-located team.

2 Co-located team members felt forced to stay at their workstations to undertake theirtasks.

3 The software enabled me to communicate with other team members whenever I neededto.

4 It was useful having a choice of ways to contact other team members (e.g. by usingtext or voice).

5 The software made it easy to determine whether team members were currentlyavailable for communication.

6 The software enabled me to communicate effectively with others whilst maintainingawareness of current information.

7 The software made it easy to ensure that all team members had a similar understandingof the situation.

8 The software enabled communications to be brief and concise.9 A high number of misunderstandings and communication breakdowns occurred when

using the software.10 The application enabled me to ‘Reachback’ (i.e. to locate staff functions outside of the

traditional area of operations) to the distributed team location.11 The software enabled me to rapidly exchange key information with another individual

on a one-to-one basis.12 The software enabled me to effectively exchange information with several other team

members at the same time.13 The software enabled information to be produced and circulated, using common

formats with visual or graphic representations where appropriate.14 The software made it easy to check that information was received and understood as

intended.15 The software enabled all team members to interact with one another as required.16 The means of electronically meeting other team members and interacting with them

was appropriate, and natural17 Ad-hoc, spontaneous interactions with other team members could take place

electronically.18 The software effectively supported social interaction (e.g. chatting or interacting

separate to the work task) with another individual on a one-to-one basis.19 The software effectively supported social interaction with several other team members

at the same time.20 The software was better at helping team members to exchange information and facts

relevant to the task than in helping them get to know each other as colleagues.21 The software enabled all members of the team to undertake their roles and

responsibilities.22 The software helped the team to synchronise their activities.23 The software made it easy to keep track of each other’s activities.24 It was easy to switch between individual tasks and team activities using the software.25 The software made it easy to manage the team’s workload effectively.26 It was easy to maintain awareness of all distributed team members through the use of

the software.27 The software made it easy to monitor distributed team members actions.

28 The software made it easy to know when to provide feedback to team members whowere not in the same room.

29 The software made it easy to identify when distributed members of the team requiredassistance and support.

30 The software made it easy to identify and deal with problems within the distributedteam.

31 The application allowed me to ‘Virtually Co-locate’ with team members in the otherroom i.e. to interact with distributed team members as if physically co-located withthem.

32 I could as easily electronically ‘meet’ the team members located in the other room as Icould meet my co-located colleagues in the same room.

33 Overall, I experienced a sense of ‘shared meeting space’ with other team memberswhen using the software.

34 Overall, I felt that the software supported me in feeling present in the distributed team.35 The software enabled me to feel a member of the distributed team.36 The software enabled me to feel familiar with my distributed team members.

37 The software aided the creation of a continuing sense of common team purpose.38 The software made it easy to build and maintain a good team atmosphere.39 The software enabled the distributed team to create and maintain an atmosphere of

trust.40 The software is a useful ‘proof of concept’.41 The software should be developed further for military use.

A1.3 The Technology Questionnaire

QUESTIONNUMBER Questions

1 The training I received on Groove was adequate to conduct the experiment.(Condition 1 only)

2 Overall, I found the software easy to use.3 I liked the software and enjoyed using it.4 The software helped me to do teamworking in an efficient and timely manner.

5 It was straightforward to learn and become familiar with the software.6 I had to frequently ask the advice of technical staff on how to use the system.7 The system behaved in a consistent way enabling me to understand how it worked.8 The system enabled easy access to common documents.9 The system enabled remote briefings to be conducted.10 System functions were well integrated.11 Few actions were required to perform functions.12 The interface enabled me to efficiently access information.13 Contacting individual users was quick and simple.14 I used the voice chat facility frequently15 The voice chat facility was easy to use.16 I felt that the voice chat facility was effective.17 I used the text chat facility frequently.18 The text chat facility was easy to use.19 I felt that the text chat facility was effective.20 I used the discussion tool frequently.21 The discussion tool was easy to use.22 I felt that the discussion tool was effective.23 I used the sketchpad tool frequently.24 The sketchpad tool was easy to use.25 I felt that the sketchpad tool was effective.26 I used the instant messaging tool frequently.

27 The instant messaging tool was easy to use.28 I felt that the instant messaging tool was effective.29 I used the electronic filing system frequently.30 The electronic filing system was easy to use.31 I felt that the electronic filing system was effective.32 I used the contacts tool frequently.33 The contacts tool was easy to use.34 I felt that the contacts tool was effective.35 I used the notepad tool frequently.36 The notepad tool was easy to use.37 I felt that the notepad tool was effective.38 The training I received on CUSeeMe was adequate to conduct the experiment.

(Condition 2 only).39 I used the video conferencing facility frequently (Condition 2 only).40 The video conferencing facility was easy to use (Condition 2 only).41 I felt that the video conferencing facility was effective Condition 2 only).42 The training I received on REMOTE was adequate to conduct the experiment.

(Condition 3 only).43 The virtual environment was natural and intuitive. (Condition 3 only).44 The avatars were effective for interacting with distributed members of the team.

Condition 3 only).45 The avatars were effective for interacting with objects in the virtual environment (e.g.

filing cabinets and post box). (Condition 3 only).

Appendix 2: The debrief prompt sheet

EXP3 - VCL debrief sheet (PJG 21_2_03)

Debrief sessions will occur after electronically completing the questionnaires for eachexperimental condition. This sheet is intended as an ‘aide memoire’ to structure the main themesto be covered in the debriefs.

Five ‘open’ style interview questions are given. Questions with brief ‘Yes / No’ answers are to beavoided. These are not intended to be prescriptive. Priority should be given to asking about theissues and problems that will have surfaced during the experimental condition.

Finally, a number of non-directive questioning and ‘prompt’ techniques have been included assuggested ways to keep the debrief discussions flowing.1. Debrief Topics for explorationALL CONDITIONS (1, 2 and 3)Q1. Were there any particular issues, problems or successes that emergedduring the last condition? (e.g. concerning the scenario, SOP’s, EXCON,software / technologies, teamworking issues – local co-located team vs.distributed team?)(Can you say more? What did you particularly like / dislike / feel was missingduring the condition?)

Q2. Did the software help you in carrying out your task activities?(In what way ? / Can you say more? / Can you think of specific examples?)

Q3. Did the software help you in interacting with and getting to know other teammembers?(In what way? / Can you say more? / Can you think of specific examples?)

Suggestion - probe specific software applications and features:• Groove (voice chat, text chat, document tool, sketch pad tool, instant

messaging facility, electronic filing system, contacts tool, notepad tool)CONDITION 2 ONLY (CU-SEE ME VIDEO CONFERENCING)Q4a. Did the CU-SeeMe video conferencing enable you to accomplish objectivesthat you couldn’t achieve using Groove alone?(What were they? / Can you think of specific examples?)

Suggestion - probe specific software applications and features:• CUseeMe desktop video conferencing (1:1, 1: many)CONDITION 3 ONLY (REMOTE VIRTUAL ENVIRONMENT)Q4b. Did the REMOTE virtual environment enable you to accomplish objectivesthat you couldn’t achieve using Groove alone?(What were they? / Can you think of specific examples?)

Q5. How did REMOTE with its virtual environment and avatars compare with theprevious CU-SeeMe video conferencing condition?(What were the relative strengths and weaknesses / advantages anddisadvantages?)

Suggestion -probe specific software applications and features:• REMOTE (overall virtual environment, avatars – presence / location / state,

avatar-avatar trigger of chat, avatar-object trigger of Groove tools)ALL CONDITIONS (1, 2 and 3)Q6. Is there anything else we forgot to ask, that you’d like to mention?(Catch all – last chance to elicit any remaining issues).2. Example ‘open’ style guidance questions for ‘probing’ topics1. Concerning X, what were your thoughts / overall impressions?2. What issues became evident with X?3. What did you think of the specific feature / aspect of X?4. What were the strengths / what worked well / what did you like?5. What were the weaknesses / what didn’t work well / what did you dislike?6. What was missing / what additions were needed / what would you like to see

added in the future?7. Other specifics (what / when / who / where / how? )3. Prompts – to keep the discussion goingI see. That is interesting. Can you say more? Did that always occur? Can yougive me an example, a ‘for instance’, when that occurred? In what way was that a

problem for you? Broadening to everyone in the team, did you all think that wasan issue for the team?Etc etc.

Appendix 3 – The Human factors ‘crib sheet’ using during observation

Appendix 4: Observer data analysis coding table

Abbrev Definitionusr_str User identified strengthsusr_weak User identified weaknessesusr_ftr User identified areas for future developmentusr_cmnt User commentsapp/funct Functionality of the underlying applicationsco-ord Team Co-ordination and Interaction.Tm_clim Team Climatead_be Adaptive Behavioursmon/fed Performance Monitoring and Feedbackunder Understandingcomm Communicationproc/mow Planning process issues or methods of workinglead Leadershippos Productivity of Staffinfo_shr Information Sharing / Managementhci Human Computer Interface Design Issuespres Sense of Shared Presence and Awareness of Others.train Training Issuesergo/phys Ergonomics or Physical Issuestech Other Techical Issuesease_use Ease of Usescenario Scenario Issuescrash System Crashesz_question Observers’ Questions.