The Effects ofThe Effects of F Feedback on eedback on the Accuracy of the Accuracy of

Completing Flight ChecklistsCompleting Flight Checklists

William RantzWilliam Rantz

Western Michigan UniversityWestern Michigan University

November 14, 2007November 14, 2007

22

OverviewOverview Rationale & PurposeRationale & Purpose Location & DurationLocation & Duration PC-ATDPC-ATD ParticipantsParticipants Checklists & Flight PatternChecklists & Flight Pattern Dependent Variables Dependent Variables Independent Variables & Integrity of IVsIndependent Variables & Integrity of IVs Experimental DesignExperimental Design ResultsResults Discussion & Future ResearchDiscussion & Future Research

33

RationaleRationale Improper use of checklistImproper use of checklist

(Degani, 1992, 2002; Diez, Boehm-(Degani, 1992, 2002; Diez, Boehm-Davis, & Holt, Davis, & Holt,

2003; Turner,2003; Turner, 2001)2001)

Most common error cited in LOSA dataMost common error cited in LOSA data Observational dataObservational data 54% of errors54% of errors

(Helmreich et al., 2001)(Helmreich et al., 2001)

Contributing factor to numerous accidentsContributing factor to numerous accidents Improper configuration of aircraftImproper configuration of aircraft

(NTSB, 1969, 1975,1982…)(NTSB, 1969, 1975,1982…)

44

PurposePurpose

To examine whether pilots would To examine whether pilots would complete airplane checklists more complete airplane checklists more accurately when they received post-accurately when they received post-flight graphic and verbal feedbackflight graphic and verbal feedback

No study in aviation has attempted to No study in aviation has attempted to increase checklist accuracy using increase checklist accuracy using experimental manipulation of IVsexperimental manipulation of IVs

55

Location & DurationLocation & Duration

The PC-ATD laboratory is located in a The PC-ATD laboratory is located in a building adjacent to WMU’s Aviation building adjacent to WMU’s Aviation Education Center in Battle Creek, MIEducation Center in Battle Creek, MI

Data collection took approximately 66 Data collection took approximately 66 sessionssessions March 20, 2007 through April 14, 2007March 20, 2007 through April 14, 2007 200 flight trials200 flight trials

66

Personal Computer-Aviation Personal Computer-Aviation Training DeviceTraining Device

77

C-172 Instrumentation PanelC-172 Instrumentation Panel

88

Observation RoomObservation Room

99

ParticipantsParticipants

8 WMU flight students8 WMU flight students 1 female & 7 males (20 – 26 years old)1 female & 7 males (20 – 26 years old) Private Pilot CertificatePrivate Pilot Certificate 125 minimum flight hours125 minimum flight hours Instrument ratedInstrument rated

1010

Main Dependent Variable Main Dependent Variable

The number of checklist items completed The number of checklist items completed correctly per flight correctly per flight

1111

Secondary Dependent Variable 1Secondary Dependent Variable 1

The percentage of total errors for each of The percentage of total errors for each of the six flight segments during each the six flight segments during each experimental phase (baseline, feedback, experimental phase (baseline, feedback, and reversal) per participant and reversal) per participant

1212

Secondary Dependent Variable 2Secondary Dependent Variable 2

The percentage of baseline trials The percentage of baseline trials participants performed each of the participants performed each of the checklist items incorrectly checklist items incorrectly

1313

Experimental PhasesExperimental Phases

Baseline Baseline Only technical feedback of flight performance was givenOnly technical feedback of flight performance was given

Checklist Graphic Feedback & Vocal Praise Checklist Graphic Feedback & Vocal Praise Technical feedback of flight performance Technical feedback of flight performance Graphic feedback on the total number of checklist items Graphic feedback on the total number of checklist items

completed correctly per flight completed correctly per flight Graphic feedback on the number of items completed correctly, Graphic feedback on the number of items completed correctly,

completed incorrectly, and omitted for each of the six flight completed incorrectly, and omitted for each of the six flight segments per flight segments per flight

Vocal praise for any improvementVocal praise for any improvement

ReversalReversal Only technical feedback of flight performance was given Only technical feedback of flight performance was given

1414

Graphic Feedback:Graphic Feedback:Total Items CorrectTotal Items Correct

P13 AM Pair 3 - Items Completed Correctly

0

5

10

15

20

25

30

35

40

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Trials

Nu

mb

er o

f It

ems

1515

Graphic Feedback:Graphic Feedback:Flight Segment Flight Segment

Checklist Items Per Segment

0

5

10

15

20

25

30

35

40

Before Takeoff After Takeoff Cruise Arrival Pre Landing After Landing Total

Checklist Segment

Nu

mb

er o

f It

ems

Total Items Completed Correctly Incorrect Omitted

17 2 5 7 5 4 40

1616

Integrity of the IVIntegrity of the IV

Technical flight and checklist feedback Technical flight and checklist feedback were read from prepared scripts were read from prepared scripts

Participants were asked to initial the Participants were asked to initial the technical flight diagrams and the checklist technical flight diagrams and the checklist feedback graphs and returned to the feedback graphs and returned to the experimenter. experimenter.

Integrity of IV = 100%Integrity of IV = 100%

1717

Experimental DesignExperimental Design

A multiple baseline with reversal design A multiple baseline with reversal design across paired individualsacross paired individuals Initial phase changes occurred when Initial phase changes occurred when

performance was judged as stable upon performance was judged as stable upon visual inspection visual inspection

1818

Results Results

All participants increased performance All participants increased performance accuracy over baseline when post-flight accuracy over baseline when post-flight checklist feedback and praise was addedchecklist feedback and praise was added

Improvements in performance remained Improvements in performance remained during intervention withdrawalduring intervention withdrawal

1919

Results-Figure 1Results-Figure 1

Average percentage of checklist items Average percentage of checklist items completed correctly increased from 53% completed correctly increased from 53% during the baseline phase to 98% during during the baseline phase to 98% during the last three sessions of the intervention the last three sessions of the intervention phase phase

The average percentage of checklist items The average percentage of checklist items completed correctly was 99% during the completed correctly was 99% during the return to baseline conditionreturn to baseline condition

2020

ResultsResults

2121

ResultsResults-Figure 2-Figure 2

During all flights, 1,973 total errors were During all flights, 1,973 total errors were observedobserved

The average percentage of segment The average percentage of segment errors was highest for the after take-off errors was highest for the after take-off segment (88%, range = 71.43% - 100%)segment (88%, range = 71.43% - 100%)

The average percentage of segment The average percentage of segment errors was lowest for the pre take-off errors was lowest for the pre take-off segment (32%, range = 5.88% - 68.91%)segment (32%, range = 5.88% - 68.91%)

Please see Figure 2 hand out Please see Figure 2 hand out

2222

Results-Figure 3Results-Figure 3

Generally, the percentage of errors by Generally, the percentage of errors by flight segment varied across participants flight segment varied across participants and flight segmentsand flight segments

Errors decreased considerably for all Errors decreased considerably for all participants during intervention. participants during intervention.

Errors were very low during reversalErrors were very low during reversal Please see Figure 3 hand outPlease see Figure 3 hand out

2323

Results-Table 1Results-Table 1

Percentages that are 50% or greater are shaded for Percentages that are 50% or greater are shaded for each participant. Also, the checklist item name is each participant. Also, the checklist item name is shaded if the percentage of error was 50% or shaded if the percentage of error was 50% or greater for four or more participants greater for four or more participants

The highest frequency of errors occurred for two The highest frequency of errors occurred for two items in the after take-off segment: Checking flaps items in the after take-off segment: Checking flaps and engine instruments, with 99 errors each. and engine instruments, with 99 errors each.

The pre landing items were the second highest The pre landing items were the second highest errors errors

The after landing segment had the third highest The after landing segment had the third highest errors errors

Please see Table 1 hand outPlease see Table 1 hand out

2424

Recap of ResultsRecap of Results

Checklist errors were reduced or Checklist errors were reduced or eliminated during the intervention phaseeliminated during the intervention phase

Performance improvement maintained Performance improvement maintained during reversalduring reversal

2525

Possible Confounding VariablesPossible Confounding Variables

These variables could account for These variables could account for variability in pilot performance:variability in pilot performance: PC-ATD experience levelPC-ATD experience level Recency of flight experienceRecency of flight experience Recency of flight in aircraft typeRecency of flight in aircraft type Fatigue/stressFatigue/stress

2626

LimitationsLimitations

Limited timeline of semester Limited timeline of semester

Transferability to other simulator Transferability to other simulator platformsplatforms

Transferability to actual flight trainingTransferability to actual flight training

Partial out intervention components Partial out intervention components (graph vs vocal)(graph vs vocal)

2727

Future ResearchFuture Research

Replicating the current study during actual training flights Replicating the current study during actual training flights when flight conditions such as weather and airport traffic when flight conditions such as weather and airport traffic differ differ

Replicating the current study and ascertaining whether Replicating the current study and ascertaining whether checklist compliance transfers to actual flight checklist compliance transfers to actual flight

Determining how long gains in checklist accuracy would Determining how long gains in checklist accuracy would continue in the absence of post-flight feedback and continue in the absence of post-flight feedback and praise praise

Investigating the nature of the rule changes and whether Investigating the nature of the rule changes and whether accurate checklist use would generalize to actual flight accurate checklist use would generalize to actual flight

2828