Acquisition and Retention f T UAV C di tiof Team UAV Coordination

Skill.Skill.Nancy Cooke1,2, Nia Amazeen2, Dee Andrews3, Jasmine

Duran1 Jamie Gorman1 Harry Pedersen1 Leah Rowe1, 2, 3Duran , Jamie Gorman , Harry Pedersen , Leah Rowe , , , Amanda Taylor1,

and Jennifer Winner1,2

Cognitive Engineering Research Institute1

Arizona State University2

Air Force Research Laboratory3Air Force Research Laboratory

21-23 May 2007

CERI’s Human Factors of UAVsCERI s Human Factors of UAVs Workshop

Chandler, AZ

C ll b tCollaborators

AFRLDee Andrews , Ph.D.P t Fit ld Ph D

ASUNia Amazeen, Ph.D.

Pat Fitzgerald , Ph.D.Kevin Gluck , Ph.D.Jerry Ball , Ph.D.

Research Assistants Jasmine Duran

Jamie C. Gorman, Ph.D.CERTT Developer: Sandia ResearchSteven M. Shope , Ph.D.

Jamie C. Gorman, Ph.D.Harry K. Pedersen

Leah RoweEugene SlutskiySteven M. Shope , Ph.D. g yAmanda Taylor Jennifer Winner

Overview of TalkOverview of Talk

T C di ti d UASTeam Coordination and UAS ControlT C iti & C di tiTeam Cognition & CoordinationExperiment 1: Retention of Team C di tiCoordinationExperiment 2: Training Team C di tiCoordinationConclusions

Team Coordination and UAS Control

2 Operators: 1 UAV2 Operators: 1 UAV

Team Coordination and UAS ControlControl

Team Coordination and UAS C t lControl

TEAM COGNITIONAND

COORDINATION

Team Cognition in Practice

Interdependent groups ofInterdependent groups of individuals who plan, decide,

perceive, design, solve problems, and act as an integrated unitand act as an integrated unit.

What is team cognition? How can we measure it? How can we improve it?

Our Context: Uninhabited Air Vehicle Synthetic Task Environment

(CERTT Labo ato )(CERTT Laboratory)

In our UAV STE three operators must coordinate over

headsets in order to maneuver their UAV to take pictures of groundtake pictures of ground

targets

CERTT UAV TeamsCritical Features: Heterogeneous,Critical Features: Heterogeneous,

Interdependent

AVOPilot

Common Display

Current/Next Target Controls UAV airspeed headingPilotCurrent/Next Target

Heading

Altitude

Controls UAV airspeed, heading, and altitude and monitors air vehicle systems

PLODEM

Altitude

Airspeed

PLODEM

Sensor Operator

NavigatorControls camera settings, takes photos, and monitors camera

Plans route from target to target under constraints monitors camera

systemstarget under constraints

T C itiTeam Cognition

Cognitive activity at the team level (planning, decision making, perceiving, situation assessment)Relevant to C2 performance and pother cognitive team tasksTraining and technologicalTraining and technological interventions can facilitate

Team Cognition Team Level Perception, Attention,Team Level Perception, Attention,

Thinking, and ActionA Measure individuals

+ +

Inside the Head ViewA Measure individuals and aggregateIncreasing similarity or convergence over time is associated withis associated with better performance or implicit coordinationAssumptions of homogeneity

Team cognition as collective cognition

Interactionist ViewB

g y

It is more than the sum of the cognition of individual B

Team cognition as holistic

gteam membersIt emerges (Gestalt - like) from the interplay of the individual cognition of

h t bcognition each team member or cognitive entity

Team Cognition Team Level Perception, Attention,Team Level Perception, Attention,

Thinking, and Action

Interactionist ViewBTeam cognition is in the interactions among team membersCommunication, coordination,

h d ll f i f i

Team cognition as holistic

push-and-pull of info. is team-level cognitive processingFocused on team process (vs. team member knowledge)as holistic

cognitionteam member knowledge)Team cognition (or team mind) is observable in the team’s interactions

Results That Support H li ti T C itiHolistic Team Cognition

600

Acquisition of Team Skill

Performance and process change with team-level experience and periods of nonuse 300

400

500

600

am P

erfo

rman

ce

Tm 1Tm 2Tm 3Tm 4Tm 5Tm 6Tm 7Tm 8

Knowledge is relatively stable after training Manipulations (dispersion, knowledge sharing cross

0

100

200

1 2 3 4 5 6 7 8 9 10

Mission

Tea Tm 8

Tm 9 Tm 10Tm 11

knowledge sharing, cross training) affect knowledge, but not performanceOther factors (team familiarity, 450

500

Team of internet video gamers exceed typical performance levels on UAV task

Other factors (team familiarity, video game experience) correlate with performance, but not knowledge.

200

250

300

350

400

450

perfo

rman

ce ra

te BenchmarkExp1Exp2Exp3Exp4

150

200

1 2 3 4

UAV mission

Di l iDisclaimer

We are NOT saying that there is nothing th hil i id th h d fworthwhile inside the heads of

individuals….But we ARE saying that in our studies theBut we ARE saying that in our studies the interactions among individuals provide more predictive power (are more relevant to team performance) than individual cognition

Implications of Holistic or Interactionist View

Measures & MetricsFocus on process over knowledgeProcess-based measures e.g., team SA, coordination

Theories and ModelsTheories and ModelsInspired by ecological psychology Dynamical systems modelsDynamical systems models

InterventionsProcess vs. knowledge orientedProcess vs. knowledge oriented

Team Coordination in C d d C t lCommand-and-Control

Coordination, a team-level cognitive process,Coordination, a team level cognitive process, is a type of team cognitionTeam coordination is the timely and adaptive exchange of information among teamexchange of information among team members/cognitive entitiesIn the command-and control context, team ,coordination means getting the right information to the right person and the right timet eObservations in our lab and of C2 problems in general have identified team coordinationas central to team performanceas central to team performance

Exp. 1 Method: Experimental Design117 males(92) & females(25) divided into 39 3-person (unfamiliar) Session 2 teamsTwo between subjects conditions

3-5 weeks 10-13 weeks

Retention Interval

(retention interval and familiarity) randomly assigned with scheduling constraintsParticipants randomly assigned to one of three roles

9 Teams10 Teams

Sam

e

mpo

sitio

n

three rolesSession 1: 5 40-min missionsSession 2: 3 40-min missions

10 Teams10 Teams

Mix

edCom

i C i iS C diti Mixed Condition

Session 1 Session 2AVO PLO DEMPC AVO PLO DEMPC

Same Condition

Session 1 Session 2 AVO PLO DEMPC AVO PLO DEMPC

RetentionRetention

IntervalInterval

Exp. 1 Hypotheses

H1: Teams in the Long retention-interval condition will demonstrate performance and cognitive (knowledge, process, p g ( g , p ,coordination, SA) deficits compared to teams in the Short retention-interval condition.

H2: Mixed teams will demonstrate performance and cognitive deficits compared to teams in the Same condition.

H3: Retention interval and familiarity should interact, whereby the deleterious effects of unfamiliar team members are more severe at the short retention interval compared to the long one (team member familiarity will decline with time)

Exp. 1 Results: Team PerformancePerformance

Retention IntervalRetention Interval

Exp. 1 Results: Team PerformancePerformance

Retention IntervalRetention Interval

Exp. 1 Results: No Team Performance DecrementNo Team Performance Decrement

for Short –Same TeamsNo decrement for short-same teams (H1) 1.0

1.5

ence

Mixed teams show significant decrement after -0.5

0.0

0.5

man

ce D

iffer

eor

es Short Mixed

Long Mixed

Short Same

break (H2)

No improvement for short-same teams -2.0

-1.5

-1.0

am P

erfo

rm Sco Short Same

Long Same

short same teams-2.5

6 7 8

Mission

Te

Exp. 1 Results: Team ProcessTeam Process

Mixed teams show improved process after the break.

0.4

0.5

ore

.

MIXED

Mission x Familiarity effect

S

0.2

0.3

fere

nce

Sco

Same teams showed no process change after the interval

0

0.1

Diff after the interval

Mixed teams did not show a

SAME

-0.16 7 8

Mission

decrement and in fact, improved across missions

MEASURING &MEASURING & MODELING O G

TEAM COORDINATIONTEAM COORDINATION

Target ProcedureGoal: Take picture of a ground target

Target Procedure

AVO Information (I)

Actors Activities

DEMPC AVO

Interdependencies

PLO

DEMPC Feedback (F)

Negotiation (N) AVO PLO

PLO AllDEMPC Feedback (F) PLO All

AVO

I N FAVO

AVO AVO

PLO

DEMPCDEMPC

1 2 3AVO PLOPLO

Recording Time Stamps of the dTarget Procedure

UAV task elements

I

NFF

Intrinsic Geometry (IG) y ( )Metric of Coordination

IF −NF −

F

Landmarks + Intrinsic G i l S iGeometry = Trial Series

Landmarks = targets

IG 1

targets

(TRIAL SERIES)

Mixed Teams are More Flexible/More Exploratory

0.70.80.9

1

exib

ility

ent) Flexible

SAME

MIXED

(M = .57)

0.30.40.50.6

rdin

atio

n Fl

eH

urst

Exp

one

RandomSAME

(M = .34)

(M .57)

00.10.2

Coo

r (H Rigid

Same Mixed

Coordination Flexibility was correlated with post-retention performance decrement

(r (30) = -.39, p < .03)

Mixed Teams are More Stable To Perturbation

0.04

0.05

ity nt) SAME

0.01

0.02

0.03

atio

n St

abil

ov E

xpon

en

Flo

UnstableMIXED

(M 005)

(M = .025)

0 03

-0.02

-0.01

0

Coo

rdin

a(L

yapu

no Flow

Stable(M = -.005)

-0.03

Same Mixed

Coordination Stability was correlated with post-retention numberCoordination Stability was correlated with post retention number of Team SA roadblocks overcome (rho (36) = -.36, p < .03)

Summary of Experiment 1 D iDynamics

SAME MIXED

MORE STABLELESS STABLE

E i t 1 Fi diExperiment 1 Findings

Long retention intervals and changes in team composition result in short-livedteam composition result in short lived team performance decrementsHowever these same factors (especiallyHowever these same factors (especially changes in team composition) result in more adaptive (flexible stable) teamsmore adaptive (flexible, stable) teams in terms of coordination

Experiment 2 Research Question Can we devise a training protocol that mimics the dynamics of Mixed teams tomimics the dynamics of Mixed teams to produce adaptive team coordination?

Procedural: Procedural Training – rigid, prescriptive I N Fprescriptive I N F

Cross-Trained: Shared Mental Models –cross train team memberscross train team members

Perturbation: constrain possible interactions to force new patternsinteractions to force new patterns

Experiment 2 Designp g

26 3 ( f ili )26 3-person (unfamiliar) teams completed all 9 missions (six additional teams did not) Session 1 Session 2)Three training interventions (between subjects condition) 8-10 weeks

AVO PLO DEMPC AVO PLO DEMPC

10 Procedural

Nine missions (within subjects) with an 8-10 week interval separating the 5th and 6th missions

Interval8 Knowledge

8 Perturbedthe 5 and 6 missionsTeams randomly assigned to condition with constraints and

8 Perturbed

participants randomly assigned to one of three roles

Experiment 2 Dynamics ResultsKnowledge Process Perturbed

Correlations between coordination stability (Lyap.) and team performance in t t it ti dbl kresponse to team situation awareness roadblocks

Overcome Roadblock Perturbation:

Session 1 r (24) = -.40, p < .05( ) , p

Session 2 r (24) = -.38, p < .06

Relaxation Time to Perturbation:

Session 1 r (24) = .26, p < .10

Session 2 r (24) = .41, p < .05

Experiment 2 Results:Team PerformanceTeam Performance

KnowledgeProcess

Perturbed

Experiment 2 Results:Team Performance

Perturbed advantage

Team Performance

Perturbed advantage

Knowledge

P

Process disadvantage

Process

Perturbed

CONCLUSIONSCONCLUSIONS

ConclusionsConclusions

TheoreticalProcess view of team cognitiongFocus on interactions over knowledgeEcological, Gibsonian

EmpiricalImportance of team process over knowledgeSh t t di d t l t ti d tShort term dips due to long retention and team mixing, but adaptive in long termSuggests perturbation training for adaptivitySuggests perturbation training for adaptivity

ConclusionsConclusions

MethodologicalTeam cognition metricsTeam cognition metricsSA, Coordination, CommunicationFacilitates on-line measurementFacilitates on line measurement

ModelingDynamical Systems predictionsy y pNext steps…ACT-R AVO Agent

Thank You!

Contact:Web Sites:

cerici orgContact:

Nancy Cooke

www.cerici.orgwww.certt.com

ncooke@cerici.org Thanks to sponsors AFOSR andAFOSR and

AFRL

BACK-UPSBACK UPS

Cognitive Engineering Research Institute

Independent OrganizationNot-for-profit A partnership of

Academia

ASU PolytechnicOne of 4 distributed ASU

campusesAcademiaGovernmentIndustry

Mesa, AZ

campusesFormerly Williams AFBApplied PsychologyMesa, AZMesa, AZ

Sponsors

Air Force Office of Scientific ResearchAir Force Office of Scientific ResearchAir Force Research LaboratoryOffice of Naval Research

US 2004 Olympic Basketball TTeam

"We still have a couple of pdays, but I don't know where we are," replied USA head coach Larry Brown to a questionBrown to a question Wednesday on where his team was in its preparations. "We have p pgood moments and bad, but I've got a pretty good understanding of who needs to play Now the jobneeds to play. Now the job is to get an understanding of how we have to play."

US 1980 Olympic Hockey Team

Team Cognition ProblemsTeam Cognition Problems

INCIDENTSVincennes 1989Challenger & Columbia 9/11 Response 2001Friendly Fire IncidentsyResponse to Katrina 2005Remotely Operated Vehicles mishapsVehicles mishaps

T C iti P blTeam Cognition Problems

CHALLENGESAutomationComplex dynamic systemsComplex dynamic systemsFatigue/experienceFast TempoUncertain informationUncertain information

And…Distributed teamsAd hoc teamsAd hoc teamsInteragency coordinationCommunication needs

No Performance Decrement for Short-Same

575 2

350

425

500

orm

ance

0

1

2

Diff

eren

ce S

core

)

200

275

350

Per

fo Short-SameLong-SameShort-MixedLong-Mixed

-3

-2

-1

sqrt

(Per

form

ance

Short-Same

Long-Same

Short-Mixed

Long-Mixed

1 2 3 4 5Mission

6 7 8Mission

Team Situation Awareness MetricMetric

Inside the Head

How many targets have you photographed?

Measuring Team Situation Awareness

( di d f hCAST (Coordinated Awareness of the Situation by Team) Metric

P i d fi t h d P i d fi t h d

PLO AVO DEMPC PLO

Perceived first-hand:

Coordinated perception:

AVO

AVO DEMPC

Perceived first-hand:

Coordinated perception:

AVO

X X X

X X

DEMPC

AVO

DEMPC PLO

Coordinated action:

AVO

PLO

Coordinated action:

X X

AVO

DEMPC PLO

AVO

DEMPC PLO X

X

X

X

X X

Overcome roadblock? Overcome roadblock? YES NO YES NO XX

Exp. 1 Results: Team Situation Awareness

Teams responded to CAST roadblocks (1 per mission; change that requires coordinated response)Events associated with coordinated perception and action are scoredscoredOptimal degree to which team members are involved in coordinationScore based on match between observed coordination and optimal model (hits and false alarms)

Exp. 1 Results: pTeam Situation Awareness

00.05

Scor

es Short MixedLong MixedShort Same

Overall and continued decrease in false alarm rate

-0.15-0.1

-0.05

ffere

nce

S

Long Samein false alarm rate after the break (reduction in unnecessary

-0.3-0.25-0.2

6 7 8

F.A

. Dif MIXED interactions)

Mission x Composition x6 7 8

Mission

Composition x Retention Interval effect

M id fMost evident for Long-Mixed teams

COMMUNICATIONCOMMUNICATION ANALYSISS S

Team Communication Metrics(ONR Sponsor)

Communication is one form of team coordination Real time embedded in the taskReal-time, embedded in the taskObservable; Think aloud “in the wild”Rich multidimensional (amount flowRich, multidimensional (amount, flow, speech acts, content)Reflects team cognition at the holistic level;Reflects team cognition at the holistic level; for us this is team cognitionAutomated methods needed to exploit data prichness and accelerate data analysis

Automating Communication AnalysisAnalysis

Communication flow data captured by Comlog toolI t l ti d f h i t lki tInterval time record of who is talking to whom, when, and for how long (8 Hz)

Logged automatically by intercom system and Comlog software

Comlog analyzed by FAUCET methods

Communication content captured on audio tapeDiscourse on audio tape is manually transcribed

Transcripts are analyzed using Latent Semantic Analysis (LSA)

ProNet-- Procedural Networks

Specific Patterns: Conflicts

BEFORE AFTERPb

Pbeg

Pend AendAendPend

Pbeg

Pend Aend

DbegDbeg AbegAbegDbeg

DendDend

Dend

C i ti R ltCommunication Results

Distinguish high from low performing UAV teams

Co located vs distributedCo-located vs. distributedLow vs. high workload

Content coding of macrocognitive processes R2 52R2=.52.Enron email: unique chains detected surrounding Enron-critical eventssurrounding Enron critical eventsExploring patterns in chat dataUse for coordination coding

SummarySummary

Long retention intervals and mixing causeLong retention intervals and mixing cause temporary hiccups in team performance (H1, H2)However, there may be long-term benefits to these perturbations of coordination in terms ofthese perturbations of coordination in terms of adaptive, flexible teams (better process, team SA).C di ti t i i di t th t di tiCoordination metric indicates that coordination variability/stability may be most predictive of performance.Possible tradeoff between immediate performance and long-term adaptability

SSummary

Model provides a qualitative look at team coordination differences. Same teams achieve selfSame teams achieve self-organization, but not mixed teams.Implications for improving team

+p p g

coordination through training interventionsCoordination metric has generalCoordination metric has general appeal for team assessment (especially if automated through communication analysis)communication analysis)

SSummary

Perturbed training resulted in better performance in novel/high workload missions. Process - poorest performance.pInteresting relationships between flexibility/stability/adaptabilityAdaptive teams may suffer some performance loss along theAdaptive teams may suffer some performance loss along the way, but may be stronger in the endTeam training can focus on

Rigid procedural training (PROCESS) rigid teamRigid procedural training (PROCESS) rigid teamor

Presentation of deliberate and varied coordination experiences (PERTURBED) adaptive team

Coordination EventsCoordination EventsIdentified coordination events of good and bad teamsIdentified coordination events of good and bad teamsDeveloped coordination logger

Experimenter–recorded observations of information passing at each waypointpassing at each waypoint Examples:

AVO was told restrictionsPLO was told effective radiusPLO was told effective radiusAVO/PLO coordinated target airspeed

Record message present/absent, sender/ receiver repetition timingrepetition, timing10% of missions were coded by two raters; Weighted Kappa =.675

Eventually automate using communication metricsEventually automate using communication metrics