Effective Team Support forTime Critical ComplexWork Environments

PIs:Bonnie E.John, Katia Sycara

Key personnel:Sandy Esch, Joseph Giampapa, Rahul Singh

7mar03 2

Goals of the project

• Advance understanding of team performance• Provide a simulation environment to cognitive modeling /

teamwork research community– Executables and source code

– Operated by humans and computational cognitive models

– Scenario editor

– Data logging

– Documentation

– Data of team performance

• Build cognitive models of team members• Build software agents to support team members & the team• Evaluate effectiveness of approach and resulting systems

7mar03 3

Outline of the talk

• Domain of interest• Simulation environment overview• Teamwork characteristics of the simulation

environment– Factors effecting performance of teams– How these factors can be manipulated in the simulation

environment– Dependent measures of team performance

• Project status and next steps

7mar03 4

Range operations

• Safe and efficient operation of the launch range during launch

• Several distributed range operators must collaborate to achieve successful launch within the launch window or abort the mission – Responsible for monitoring a particular area in the launch zone– Negotiate with other range operators for resource allocation

• Some tasks include…– Maintaining clear airspace over the entire range during launch– Predicting plume dispersion– Efficient allocation of resources to minimize risk and cost while

maximizing safety

• Typically a team task with distributed responsibility• Typically a time critical task• Information-rich and communications-intensive

7mar03 5

Simulation environment forcognitive modeling / teamwork research:

The MORSE task

MORSE Architecture

• Flow of the task is controlled by the MORSE Command• MORSE Command models entire mission and simulation world• MORSE Stations display focused subset of simulation world to each user• MORSE Command maintains accounting of costs incurred, coordination penalties

MORSE-Command

MORSE-Station CharlieAscension Islands

MORSE-Station Bravo

Antigua

MORSE-Station Alpha

Cape Canaveral

Scenario File

Weather Queries

Timing Synchronizatio

n

Incursion Information

Shared Information

between Stations

7mar03 7

Team performance(Galbraith, 1977; Kraut, Fussell, Lerch, & Espinosa, 2002)

• The more uncertainty in a task, the more information-processing necessary to achieve a given level of performance (Galbraith, Organization Design,1977)

• The more coordination within a team, the higher level of performance achieved (Kraut, et. al., 2002)

• Thus, to increase performance on a given task, take measures to – decrease uncertainty

– make information-processing less costly

– improve coordination

• Such measures may include– Designing the task environment, including software agents

– Designing the organization

7mar03 8

Team performance(Galbraith, 1977; Kraut, Fussell, Lerch, & Espinosa, 2002)

• The more uncertainty in a task, the more information-processing necessary to achieve a given level of performance (Galbraith, Organization Design,1977)

• The more coordination within a team, the higher level of performance achieved (Kraut, et. al., 2002)

• Thus, to increase performance on a given task, take measures to – decrease uncertainty

– make information-processing less costly

– improve coordination

• Such measures may include– Designing the task environment, including software agents

– Designing the organization

7mar03 9

Uncertainty or information to be acquired and processed

Amount of information possessed by organization

Amount of information required for task

performance

Determined by goal diversity,level of goal performance, division of labor

Determinants of informationand task uncertainty

(adapted from Figure 3.1 Galbraith, 1977)

7mar03 10

Factors affecting uncertainty (Galbraith 1977) instantiated in MORSE

• Diversity of goals– Safe launch

• No incursions or interceptors within Impact Lines• Toxic plume would not disperse to populated area

• Level of goal performance– Time constraints– Limited resources: interceptors, weather balloons,

participant attention & actions– Cost metrics

• Division of labor– Small team, organically coordinated, or trained

• Territory covered by each team member• Radar placement

7mar03 11

Factors affecting uncertainty (Galbraith 1977) manipulable in MORSE

• Diversity of goals– Safe launch

• No incursions or interceptors within Impact Lines• Toxic plume would not disperse to populated area

• Level of goal performance– Time constraints– Limited resources: interceptors, weather balloons,

participant attention & actions– Cost metrics

• Division of labor– Small team, organically coordinated, or trained

• Territory covered by each team member• Radar placement

7mar03 12

Mechanisms to economize oninformation processing capacity (Galbraith 1977)

instantiated in MORSE

• Rules, programs or procedures - decisions in advance of their execution– Could arise organically, or could be trained

• Discretion guided by planning or professionalism– Could plan via chat (or talking outside the experiment)– Could involve activation of radar, activation/location of

interceptors

• Adjusting the hierarchy of authority (Narrowing span of control)– Could arise organically, or could be trained– Could assign roles, e.g., one person monitor weather,

others handle incursions

7mar03 13

Mechanisms to economize oninformation processing capacity (Galbraith 1977)

manipulable in MORSE

• Rules, programs or procedures - decisions in advance of their execution– Could arise organically, or could be trained

• Discretion guided by planning or professionalism– Could plan via chat (or talking outside the experiment)– Could involve activation of radar, activation/location of

interceptors

• Adjusting the hierarchy of authority (Narrowing span of control)– Could arise organically, or could be trained– Could assign roles, e.g., one person monitor weather,

others handle incursions

7mar03 14

Reduce the need for information processing (Galbraith 1977 )

manipulable in MORSE

• Environmental management– Provide software agents that, in effect, change the

environment

• Creation of slack resources– Evaluation function, e.g., tolerate some incursions or

interceptors left within the Impact Lines, change the cost of deploying weather balloons or sharing data, etc.

• Creation of self-contained tasks– Could arise organically, or could be trained– E.g., handle only own radar areas, no communication

7mar03 15

Increase capacity to process information (Galbraith 1977)

manipulable in MORSE

• Invest in vertical information systems– Provide software agents to collect and process

information for the team

• Creation of lateral relations– Small team, free to communicate; this mechanism does

not apply to this task

“The organization must adopt at least one of the five strategies when faced with greater uncertainty. If it does not consciously chose one of the five, then slack, reduced performance standards will automatically happen.” (p. 55, Galbraith 1977)

7mar03 16

Coordination in teams(Kraut et. al. 2002)

• Factors contributing to coordination – Communication frequency– Communication evenness– Common history– Shared cognition

• Factors are beneficial individually, compensate for each other

• The Morse environment can measure these factors more directly than Kraut, et. al. were able to in their study of a simulated management game

7mar03 17

Objective Measures in MORSE

• Communication frequency– Clicks to ask for resources– Chat messages

• Communication eveness– Who asked whom for what– Who chatted with whom about what

• Common history– Time on task

7mar03 18

Shared cognition

• Kraut, et. al., investigated shared cognition about expertise of team members– Skills will not differ between team members in MORSE– Software agents will have specialized expertise that would

complement what the people are doing

• Other types of shared cognition to investigate– Weather (what is it doing and what does everyone else

know about it?)– Plume dispersion– Incursions on the way– Availability of other participants (how busy are they)

• Measuring shared cognition in MORSE– Questionnaires after a trial (like Kraut et. al. did)– Think-aloud protocols during a trial

7mar03 19

Coordination in the MORSE task

• Mechanistic coordination vs. organic coordination– Mechanistic: align individual actions through structure o

directive– Organic: individuals exchange information about their

current states and adjust their behavior to other’s goals and actions

– Could have either, or both, in MORSE

• Benefits of coordination– Allocate limited or costly resources more efficiently– Improve performance (according to Kraut, et. al.)

• Costs of coordination– It takes time away from other task activities

7mar03 20

Direct measures of coordination in MORSE

• Failures, e.g.,– Two participants assign interceptors for same incursion– Two participants deploy weather balloons to same

location– Denial of resource request

• may be failure from one participant’s perspective• may not be a failure for the team

– Using a costly interceptor when coordinating would have been less costly

– Chat to plan, then plan ignored– Launch safety disagreement

• Successes, e.g.,– Clicks and acceptances to share/release resources– Chat to plan, then plans executed

7mar03 21

Measures of performance

• Both team and individual• Evaluation function

– Launch safety miscalculation– Incursions or interceptors left with Impact Lines– Radar costs– Interceptor costs– Weather balloon costs

7mar03 22

MORSE status

• Interface– Colors accommodate color-blind population– Icons encode speed, direction, ownership– Less cluttered screen– More natural ways to provide information– Window management– Chat

• Evaluation function– Drafted, not yet implemented

• Far fewer bugs!

7mar03 23

Next steps

• Building MORSE – Redesign command sequences for less clicking– Usability of scenario editor– Usability of data logging– Documentation

• Collect human performance data• Develop cognitive models• Develop software agents• Study their effectiveness

7mar03 24

Demo!

• In the downstairs lobby (right outside the doors)at the break and after the sessions