elizabeth “scottie-beth” fleming committee members: dr. amy pritchett, chair dr. karen feigh

Elizabeth “Scottie-Beth” Fleming

Committee Members:Dr. Amy Pritchett, Chair

Dr. Karen FeighDr. Ute Fischer

Sponsored by the FAA, Tom McCloy as Technical Monitor

DEVELOPING A TRAINING PROGRAM FOR THE TRAFFIC ALERT AND COLLISION

AVOIDANCE SYSTEM IN CONTEXT

2

Overview

+ Introduction to TCAS and Background Information+ Design of Training Program+ Evaluation of the Training Program+ Results+ Conclusions

Develop a training program intended to improve pilots’ understanding of TCAS use for collision avoidance in a

range of traffic situations

3

Traffic alert and Collision Avoidance System (TCAS)

+ Pilot always shown the Traffic Situation Display (TSD)+ TCAS delivers a two stage advisory and vertical avoidance maneuver

Traffic Advisory (TA) - ‘Traffic Traffic’ Resolution Advisory (RA) - ‘Climb Climb’

+ Pilot is to follow an RA, even if it conflicts with ATC instructions, unless the pilot believes that safe flight would be jeopardized

Federal Aviation Administration (2000). Introduction to TCAS II Version 7. Washington, D.C.

4

Collision Avoidance in a Broader Context

Pilots don’t only interact with TCAS, they also receive information from ATC and the environment

Pilot’s ResponseAdvised Maneuvers for Collision Avoidance

Notification and Awareness

Awareness of other aircraft via TSD

TCAS Traffic Advisory

TCAS Resolution Advisory

ATC Traffic Callout

Visual acquisition of other aircraft

ATC Maneuver

Compliance to TCAS

Compliance to ATC

Personal Assessment and Maneuver

DecisionParty-line Information

5

Current TCAS Training Requirements: Ground Training

+ Classroom or computer based + >60 training requirements+ Measures performance through

quizzes and/or activities

To comply to the RA, you should

Pull the stick back

Push the stick forwardFederal Aviation Administration (2001). Advisory Circular No. 120-55B: Air Carrier Operational Approval and Use of TCAS II. Washington, D.C.

6

Current TCAS Training Requirements: Flight Training

7

Observed TCAS Use and Operation

o The TCAS Operational Performance Assessment (TOPA) monitored the terminal area of 8 major airports and examined pilot compliance to climb and descend advisories

41% to Climb RA’s

59% to Descend RA’s

TOPA observed compliance rates of

Misunderstanding/ Confusion

Aggressiveness

Non-Compliance

Olson, W. and J. Olszta (2010). TCAS Operational Performance Assessment in the U.S. National Airspace. IEEE/AIAA Digital Avionics Systems Conference.

Olszta, J., & Olson, W. (2011). Characterization and Analysis of Traffic Alert and Collision Avoidance Resolution Advisories Resulting for 500' and 1,000' Vertical Separation. Paper presented at the Ninth USA/Europe Air Traffic Management Research and Development Seminar (ATM 2011), Berlin, Germany.

8


“[My FO and I] incorrectly interpreted the red 'above' target on the VSI and responded improperly. We further reviewed the procedures, agreeing that a

person should fly 'away' from the red VSI indication, if instructed via RA.”ACN:785761, 2008

Pilot reports flying into the red region on the VSI

Non-Compliance

Aggressiveness


NASA. (2009). Aviation Safety Reporting System. Retrieved August 1, 2010: http://asrs.arc.nasa.gov/

“Descending into an airplane that is clearly descending? TCAS software clearly did not give appropriate guidance, nor did it self-correct when the

initial guidance was so clearly wrong” ACN: 854982, 2009

Pilot disagrees with advised descend RA

9


Non-Compliance


Aggressiveness

Far Eastern Air B757 Response to Descend RA (TCAS advised a descent rate of 1500 FPM)

Image copied from Lacagnina (2008). Easy Does It. Aero Safety World : http://flightsafety.org/asw/oct08/asw_oct08_p44-47.pdf?dl=1

10

Research Statement and Objectives

Develop a training program intended to improve pilots’ understanding of TCAS use for collision

avoidance in a range of traffic situations

(1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment

(2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories

(1) Train pilots to understand TCAS use for collision avoidance in the actual traffic and operational traffic environment

(2) Provide pilots with a well-rounded knowledge of different traffic situations that may result in TCAS advisories

11

Demonstration Based Training

Event Based Training

Approach to Training Design

Complement of Two Methods

12

Demonstration Based Training (DBT)

+ Computer-Based Training ~25 minutes Provides conceptual understanding of TCAS Outlines rules for compliance

+ 6 segments Introduction to TCAS Traffic Situation Display TCAS Advising Logic Traffic Advisories Resolution Advisories Example Timeline of RA Evolution

+ Mid-Training Quizzes

13

DBT: Demonstrations of RA’s

Event Based Training (EBT)

+ Presents traffic events that create the requirement to act+ Builds context and complexity into each scenario as the flight

progresses+ Uses a more realistic training environment

Fowlkes, J., Dwyer, D., Oser, R., & Salas, E. (1998). Event-Based Approach to Training (EBAT). The International Jounal of Aviation Psychology, 8(3), 209-221.

14

15

Air Traffic Transcripts

Charts & Checklists

Audio Communications (Aviation Intercom)

TCAS Alerts

Simulation Architecture

Eyetracker

PartyLine First Officer

Captain

Experimenter/Instructor

Coded Log of Flights

B747-400 Simulator (RFS)

TSDPFD

ND

PFDN

D

Touchscree

n

SideStick

ATC->TSDVGA

ATC Air Traffic Sim

ulator (TGF)

TCAS Logic

Simulator Study in Integrated Flightdeck – ATC Environment

Pritchett, A., Fleming, E., Cleveland, W., Zoetrum, J., Popescu, V., & Thakkar, D. (2012). Pilot Interaction with TCAS and Air Traffic Control. Paper presented at the 2nd International Conference on Application and Theory of Automation in Command and Control Systems (ATACCS), London.

16

Structure of Events: Descend RA Example

Training Objective Context of Event Performance Measures Feedback

Session

Accurate interpretation

of and response to TCAS

Descend RA.

Instrument Meteorological

Conditions (daytime, clouds, no winds)

ATC provides no traffic information

Conflict caused by IFR traffic enroute (most likely on departure)

RA maneuvering should not violate ATC

instructions

No conflicting ATC or party-line information

Pilot disengages autopilot and flight directors

If the pilot did not meet any

particular performance

measure, review the

correct response in

regards to that measure

Pilot responds to advisory with appropriate vertical speed

Pilot ensures vertical speed is not excessive

o Pilot notifies ATC of response to TCAS advisory as the maneuver is performed

Pilot reengages autopilot and flight directors

o Pilot notifies ATC of TCAS advisory and response after clear of conflict

Pilot returns to original clearance (if needed)

Evaluating the training program’s impact

18

+ Baseline Study+ Training Study+ 6 scenarios with 2 traffic events per scenario+ Traffic events defined by

RA type ATC information Traffic density

Overview of Study

Comparison to identify the impact of training

Overview of Study

Pre-Training Data Collection

(50 minutes)

Pre-Experiment Questionnaire

Pre-Experiment Quiz

TCAS Training Program

(50 minutes)

Introduction to TCAS

Demonstration Based Training

Short Section Quizzes

Event Based Training

Evaluating TCAS Training Program

(120 minutes)

Flight Scenarios

Post Scenario Questionnaires

Debrief(15 minutes) Post- Experiment Questionnaire

19

20

Skill-Based Behavior

Rule-Based Behavior

Knowledge-Based Behavior


Rule-Based Behavior


Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?

21

Does the training program improve pilot performance in response to TCAS advisories and increase pilot understanding of TCAS?

Decrease time to achieve compliance

Decrease time to disconnect autopilot

Reduce aggressive response features

Increase percentage of RA duration in compliance

Increase appropriate response post-Clear of Conflict

Increase understanding of TCAS

Increase trust in TCAS


Rule-Based Behavior


22

Assessing the Pilot’s Response: Skill-Based BehaviorsV

erti

cal S

peed

time5 seconds

RA Climb

2 ½ sec

TCAS weakens required

vertical rate

TCAS assumed

¼ g pull-up

Clear of Conflict

TCAS assumed constant vertical

RA rate

Autopilot Disconnect

Time

TCAS RA Maneuver

Pilot’s Response

Time Pilot First Achieves Compliance


Rule-Based Behavior


X

23

Mean Time to Comply: During Training


pMM = Significance for the mixed modelps2 = Significance of the variancepm = Significance of the means

Rule-Based Behavior


24

+ During training, no significant differences+ Post training

Autopilot disconnect time decreased Pilot response was more consistent for one event

Autopilot Disconnect: Post Training


Rule-Based Behavior


25

Impact of Training Program









Rule-Based Behavior



Rule-Based Behavior

26

Assessing the Pilot’s Response: AggressivenessV

erti

cal S

peed

time5 seconds

RA Climb

2 ½ sec


vertical rate

TCAS assumed

¼ g pull-up

Clear of Conflict


RA rate

Maximum vertical rate

Maximum vertical rate difference Vertical rate

difference

TCAS RA Maneuver

Pilot’s Response


Rule-Based Behavior


*Altitude Deviation

27

Assessing the Pilot’s Response: ComplianceV

erti

cal S

peed

Not in Compliance In Compliance

TCAS RA Maneuver

Pilot’s Response

time5 seconds

RA Climb

2 ½ sec


vertical rate

TCAS assumed

¼ g pull-up

Clear of Conflict


RA rate


Rule-Based Behavior


28

Assessing the Pilot’s Response: Return to Clearance

TCAS RA Maneuver

Pilot’s Response


RA rate

TCAS weakens required vertical rate

Clear of Conflict

Alt

itud

e

time5 seconds

RA Climb

TCAS assumed

¼ g pull-up

Cleared altitude at time of RA

Pilot holds new altitudeContacts ATC and request new clearance

or ask for further instructions

Pilot begins descent back to originally cleared altitude

May inform ATC of response to RA


Rule-Based Behavior


29

Before Training

+ 89% of the pilots did not know that an RA should cause less than 500 feet of altitude deviation

+ 56% of the pilots responded that they would hold current altitude achieved after responding to an RA, as opposed to returning to their clearance

+ When asked about airline procedures for following TCAS, all 18 pilots noted the need for complying with an RA

BUT 28% of the pilots commented compliance wasn’t necessary is there was a TCAS “malfunction” or if the RA would cause an “unsafe situation”

“[Pilots] must always comply with a TCAS RA unless [aircraft] performance is hindered (i.e. operating single

engine) or [there is] an obvious TCAS malfunction (ie you can see traffic and it is not a threat)”


Rule-Based Behavior


30

+ During training, aggressive response features decreased + Post training, same decreasing trend observed

Aggressiveness

Altitude DeviationAverage Vertical Rate Difference

Maximum Vertical Rate DifferenceMaximum Vertical Rate Difference

All decreased


Rule-Based Behavior


31

+ During training, percentage compliance decreased in training event with “Climb RA” (93.1% compared to 99.6%)

+ Post training No significant differences in means observed But, trained pilots had a more consistent response

Percentage Compliance


Rule-Based Behavior


32

Return to Clearance


Rule-Based Behavior


Pilot returns to original clearance

NoYes

33

Impact of Training Program




~ Increase percentage of RA duration in compliance





Rule-Based Behavior


Rule-Based Behavior


34

Before Training

+ 94% pilots agreed with statement “I understand TCAS maneuvers when they are issued”, but… Only 50% of the pilots correctly identified the assumptions made by TCAS

advisory logic When asked to interpret TSD symbols, only 50% of the pilots got all parts of

the associated questions correct


Rule-Based Behavior


35

“My understanding of TCAS has increased”

+ 50% pilots reported an increase in understanding TCAS logic+ 27% pilots claimed to have learned about different types of TCAS RA’s

(notably, “Crossing RA’s”)

Post Training Understanding of TCAS

“Types of RA's were not previously taught. We were taught simply to comply”

Strongly Disagreed

0

Disagreed

0

Neutral

6%

Agreed

61%

Strongly Agreed

33%


Rule-Based Behavior


36

Post Training Trust in TCAS

“I am more likely to trust TCAS after completing today’s training”

“My trust in TCAS was already at a maximum so I wouldn’t be ‘more’ likely to trust it”

Strongly Disagreed

11%

Disagreed

0

Neutral

39%

Agreed

39%

Strongly Agreed

11%


Rule-Based Behavior


37

Post-Training








38

Conclusions and Future Work

Conclusions+ Pilots may not need more training, but instead need better training

Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive

EBT structure allows for the design of purposeful training events+ Current FAA mandated TCAS training objectives may not fully reflect all

training areas Language needed to address the reduction of excessive responses to RA’s

Future Work+ What facilities and technologies would be required to implement this type

of training program?+ What implications arise when considering training design versus system

design? Would incorporating human factors considerations in initial design stages

decrease the amount of required training?

40

Conclusions and Future Work

Conclusions+ Pilots may not need more training, but instead need better training

Integrating DBT and EBT methods permits ground-based and flight training material to be more cohesive

EBT structure allows for the design of purposeful training events+ Current FAA mandated TCAS training objectives may not fully reflect all

training areas Language needed to address the reduction of excessive responses to RA’s

Future Work+ What facilities and technologies would be required to implement this type

of training program?+ What implications arise when considering training design versus system

design? Would incorporating human factors considerations in initial design stages

decrease the amount of required training?

41

Acknowledgements

+ Work sponsored by the FAA, Tom McCloy as Technical Monitor + 34 pilot participants+ Dr. Amy Pritchett+ Dr. Karen Feigh+ Dr. Ute Fischer+ Dr. Wesley Olson, MIT Lincoln Labs+ Wayne Gallo, FAA+ Roger Sultan, FAA+ Kylie Garey+ TCAS Team: William Cleveland, Vlad Popescu, Justin Mullins, Anil Bozan,

Henry Tran, Jack Ridderhof, Alyssa Whitlock, Colin Ludwig, Dhruv Thaakar, Jonathan Zoetrum, Jelle Wissink

+ CEC Lab Members

elizabeth “scottie-beth” fleming committee members: dr. amy pritchett, chair dr. karen feigh

Documents

use of tcas

tcas software

observed tcas use

analysis of traffic

stage advisory

pilot compliance

atcpersonal assessment

red region