modeling and measuring situation awareness in individuals and teams

Dynamic Decision Making Labwww.cmu.edu/ddmlab

Social and Decision Sciences DepartmentCarnegie Mellon University

1

MODELING AND MEASURING SITUATION AWARENESS IN INDIVIDUALS AND TEAMS

Cleotilde Gonzalez

In Collaboration with: Lelyn Saner, Octavio Juarez, Mica Endsley, Cheryl Bolstad, Haydee

Cuevas, and Laura Strater

2

• Computational Models of SA– Individual aspects of SA– Design aspects of SA– Organizational aspects of SA

• Measures of SA– Individual SA– Shared SA

• Conclusions

Agenda

Situation Awareness

• the Perception of the Elements in the Environment within a Volume of Time and Space,

• the Comprehension of their Meaning, and• the Projection of their Status in the Near Future. • Formation of SA influenced by:

Individual abilities Interactions with others Environment

• Integrated theory of mind: ACT-R (Anderson & Lebiere, 1998)

– Shared attention (Juarez & Gonzalez, 2003, 2004)– Learning theory (Gonzalez, Lerch & Lebiere, 2003; Gonzalez &

Lebiere, 2005)– Representation of Recognition (Gonzalez & Quesada, 2003)– Learning and decision making in dynamic systems (Gonzalez et

al., 2003; Martin, Gonzalez & Lebiere, 2004)

• Micro and Macro Cognition: Convergence and Constraints Revealed in a Qualitative Model Comparison (Lebiere, Gonzalez & Warwick, 2009)

4

Computational Cognitive Models

5

Computational Cognitive Models

6

A SA meta-architecture provided a full set of cognitive models interacting with OTB, and resulting in the “commander’s SA” (Gonzalez et al., 2004; Juarez & Gonzalez, 2003)

7

Computational Models of Design Aspects of SA (Juarez & Gonzalez, 2006)

8

• Computational Models of SA– Individual aspects of SA– Design aspects of SA– Organizational aspects of SA

• Measures of SA– Individual SA– Shared SA

• Conclusions

Agenda

Individual Measures of SA: SAGAT

• Situation Awareness Global Assessment Technique (SAGAT)• Human-in-the-loop simulation exercises• Use of SAGAT queries (from GDTA)• Stop at random times and query the user• Compare response with reality of the situation

– Examples: What is the aircraft altitude?– What is the aircraft activity in this sector (en route, inbound to airport,

outbound to airport)– Which aircraft will need a new clearance to achieve landing

requirements?• SAGAT score: accuracy of the responses

Individual SA measures, learning and working memory

• Can we learn to be aware? Effects of task practice and working memory influence situation awareness (SA) - Gonzalez & Wimisberg, 2007

• How do we measure individual SA– Queries may be answered while the simulation

display is not visible or covered (Endsley, 1995) or while the display is visible, uncovered (Durso et al., 1995).

10

Methods

• The design was a 2 x 18 mixed design. Participants were randomly assigned to one of two conditions (covered or uncovered display) and they were asked to run the simulation 18 times (trials).

• Individuals were asked to answer SA queries while the simulation was paused

• Participants took the Visual Span Test (VSPAN) (Shah & Miyake, 1996).

11

12

13

Results

• The correlation between SA scores and VSPAN decreased over time

• SA scores were higher in the uncovered condition than in the covered condition– This is due mostly to perception

• The effect of practice was significant only in the covered condition, but not in the uncovered condition

18

Summary of results relevant for individual measures of SA

Measures of Shared SA (Saner, Bolstad, Gonzalez & Cuevas, in press; Saner, Bolstad, Gonzalez & Cuevas, in preparation)

Ground TRUTH

X1X2X3X4X5X6X7

Person 1

X1 X3X4 X6X7X8X9

Person 2

X1X2 X4 X6 X8

Shared SA-the degree to which team members possess the same SA on shared SA requirements (i.e. on the information that they both need to know)

(Endsley,1995, 1995b; Endsley & Jones, 2001)

A good measure of shared SA needs to account for the ACCURACY

Shared SA

Person 1Q1

Q2Q3Q4Q5Q6Q7

Person 2Q1

Q2Q3Q4Q5Q6Q7

SimQ1

SimQ2SimQ3SimQ4SimQ5SimQ6SimQ7

Situation Awareness Global Assessment Technique (SAGAT) - Seven queries while task is stopped - Objective knowledge of situation

Score Similarity = 1-absolute value of [(p1-p2)/(p1+p2)]Range from 0 to 1

A good measure of shared SA needs to account for the SIMILARITY

Method• Training at Joint Personnel Recovery Agency (JPRA) - JFCOM• 16 servicemen, 3 DoD contractors; Age M=33.85• Randomly assigned to one of four Teams:

– Navy, Army, Special Operations, or Joint Service• Utilized Cross-Training

– Five scenarios over 3 days – Each scenario had 3 to 12 incidents– Scenarios randomly stopped 3 times for SAGAT,

Communication, and Workload measures– Received training prior to the exercise

Methods and Procedure

Joint Service Cell

(p1, p2, p3, p4)

Special Operations Cell

(p13, p14, p15, p16, p17)

Army Cell

(p5, p6, p7, p8)

Navy Cell

(p9, p10, p11, p12)

• Joint Personnel Recovery Agency (JPRA) training exercise

• Four team groups (i.e. cells)• Five Predictors of Shared SA

– Experience Similarity- years in real service

– Shared JPRA Knowledge- prior experience with recovery operations

– Shared Cognitive Workload- subjective ratings, five NASA-TLX scales

– Communication Distance- inverse frequency of communication

– Organizational Hub Distance- degree of dissociation from Joint Service Cell

Possible Models

Classic Hierarchy Expected

Results

Regression Models of True Shared SA

True

Shared SA

F Adj.

R2

Constant Experience

Similarity

Shared

Knowledge

Workload

Similarity

Organizational

Hub Distance

Communication

Distance

OVERALL 5.11** .21 -.03 .09 .26* .08 .50** -.18

Scenario 1 2.56* .09 .02 -.07 -.02 .18 -.26* -.08

Scenario 2 1.55 .03 .39 .04 -.05 -.19 -.26* .00

Scenario 3 1.66 .05 .10 .19 .09 .04 -.26* -.02

Scenario 4 2.62* .11 .26 .08 .31* -.03 -.17 -.06

Scenario 5 5.79* .24 .42 .11 -.19 -.16 .45* -.24*

*p < .0 5

**p < .0 1

Conclusions – Measures of Shared SA

• Development of a Shared SA measure must account for both, accuracy and similarity of SA between members of an organization

• As shared knowledge increased, so did shared SA. • Organizational Hub Distance (OHD) is key

predictor– Physical Distance and Joint Cell Membership

• Unexpected Role of OHD– Participants processed new information directly

Possible Models

ExpectedObserved

We observed that being in branch cells was associated with higher SSA rather than being in the joint cell

27

• The success of Computational Models of SA, depends on appropriate and robust measures of individual and shared SA– Although individual measures and procedures exist, there is a

huge need for defining the methods and procedures for measuring SA at the team level

• We investigated measures of SA at both, the individual and team levels– We created a shared SA measure that builds on individual SA

• Computational models of both, SA and SSA can incorporate these measures.

Conclusions