advanced federal aviation qualification program · qualification program what it is and how to ......

Presented to: 1st Brazilian AQP Seminar

By: Dr. Douglas Farrow, FAA AQP Manager

Date: November 24, 2015

Federal Aviation Administration

Advanced

Qualification Program

What it is and How to

Regulate it

2 Federal Aviation Administration

2

The Two Questions

• What is the Advanced Qualification

Program?

– To the Regulator

– To the Airline

• What is the best way to Regulate the

Program?

– Field Office

– Regional Office

– Headquarters Office


3

AQP by the Numbers

• The AQP rule is 25 years old

• There are 30 pilot and 8 inflight certificates

• Pilots, flight attendants and dispatchers

began AQP training in:

– 1994 United pilots (21 years)

– 2006 Delta flight attendants (9 years)

– 2010 USAirways dispatchers (now in hibernation)

• Today 90% of US pilots and 70 % of US

flight attendants are in AQP


4 4

Airline Participation (Pilot)

Size Pilots Airlines Airlines Classic AQP

Large 1,000 + 22% 18 0 18

Medium 500-999 10% 08 1 07

Small 1-499 68% 55 50 05

Total 100% 81 51 30


5 5

Airline Participation (Inflight)

Size Pilots Airlines Airlines Classic AQP

Large 1,000 + 22% 18 10 07

Medium 500-999 10% 08 8 01

Small 1-499 68% 55 55 00

Total 100% 81 73 08


6

Individual Pilots & Flight Attendants

FAA Safety Performance Analysis System (SPAS) January 2015

Individual

Students

AQP Traditional Percent

Pilots

77,831

70,628

7,203

90.74%

Flight

Attendants

107,032

73,729

33,303

68.88%


7 7

Overview

• What is the Advanced Qualification Program? – Regulatory Perspective (FAA)

– Methodological Perspective (Airline)

• The 3 D’s (AQP) versus the 3 T’s (Traditional) – Documents --Tasks

– Data --Topics

– Dialogue --Times

• Comparisons with Traditional Training


8

Goal #1 for AQP

“Provide for a Special Federal Regulation

(SFAR) and Advisory Circular to permit

development of innovative training

programs.” Joint Government Industry Task Force

June 8, 1988


9 9

The Regulatory Perspective

Two Independent Training Rules

• AQP is a voluntary alternative to traditional

training.

– Chapter14 CFR, Part 121, Appendix N & O

• Traditional

– Chapter14 CFR, Part 121, Subpart Y

• AQP

• AQP allows alternative means of

compliance (alternative risk controls) for:

– Qualification (Parts 61, 63 & 65)

– Training (Parts 121, 135, 142)


10 10

The Regulatory Perspective

Defining AQP

• “…this subpart provides for the approval of an

alternative method …for qualifying, training,

certifying and otherwise ensuring the competency

of crewmembers, dispatchers, other operations

personnel, instructors, and evaluators who are

required to be trained under parts 121 and 135 of

this chapter.” » Part 121, Subpart Y, 121.901


11 11

Three Generations of AQP

1. Alternative training program (1990-2000)

1. Driven primarily by training data

2. Voluntary safety program (2000-2010)

1. Driven by data from training & reporting programs

3. Training component of an SMS (2010- )

1. Driven by data from training, safety and operations

data


12

AQP Linecheck

Pilot

Grades

Behavior Change

Generation 1.0

Operating Procedures

Operationally Realistic

Scenario-Based

Training


13

AQP SP Data

Pilot

Grades

Behavior Change

Generation 2.0



Scenario-Based

Training


14

AQP SMS Data

FA and

Pilot

Behavior Change

Generation 3.0



Scenario-Based

Training


15 15

Objectives of AQP

• To improve safety through continuously improving training and evaluation.

• To be responsive to continuing changes in industry, including

– New aircraft technology,

– Changing pilot demographics,

– Changing operational environments, and

– New training methods and equipment.

• To be responsive to continuing changes and best practices relative to

– Training &

– Evaluation

• Integration of CRM/TEM/RRM and piloting technical skills in training and evaluation.

• Data driven approach to quality control

– Training & operational data


16 16

AQP is

• Data driven

– Safety, operational & training data

• Proficiency based

– Measurement focus shifts from training to learning

– From hours to proficiency measures

• Consistent with the most recent ICAO training

guidance

– New ICAO training guidance is less prescriptive, more tailored,

more scenario-based, more CRM oriented, competency-based

versus task-based: more AQP-like

» Evidence Based Training (EBT), Advanced Training &

Qualification Program (ATQP)


17 17

AQP is

• Designed for innovation • Flexibility in content, strategy, duration, interval, media, etc.

• Subpart Y (AQP) is a process rule, Subparts N & O (Traditional)

are product rules

• Established goal of continuous improvement vs. compliance

• Facilitates alternative means of compliance by

– Requiring extensive front end analysis to justify training footprint

– Review & approval by FAA extended review team (ERT)

» Headquarters & Field Personnel

– Requiring continuous submission of data to justify not only the

deviation, but also those elements that comply with regulations


18 18

The Methodological Perspective

• Replaces one-size-fits-all FAA-generated

training with airline-generated training

tailored to each fleet at each carrier, based

on analysis and continuous data collection. • Training data

• Operational data

• Shifts the focus from teaching (hours) to

learning (competencies).

• Uses explicit Instructional Systems

Development model.


19 19

The Methodological Perspective

• Frames training regulations as risk controls.

• Allows alternative risk controls with

approval of FAA Extended Review Team: • Principle Operations Inspector (POI)

• FAA Program Manager, AQP

• All traditional training regulations apply,

unless a specific alternative means of

compliance is justified and authorized.


20 20

AQP includes Front End Analysis

(FEA)

• Job task analysis

• Competency analysis

• Student entry analysis

• Criticality/Currency analysis

– 2 x 2 Training Risk Matrix

• Media & Methods analysis

• Qualification Standards Development


21 21

Plus continuous data collection and

analysis

• Traditional Measures (pilot recurrent):

– 2 pass/fail gates

• AQP Measures (pilot recurrent) on average:

– 2 pass/fail gates + 1 validation gate

– 50 measured items (maneuvers, etc.) on a 4 or 5

point scale

– 3-4 pretest/posttest measured items


22 22

Overview

• What is the Advanced Qualification

Program?

– Regulatory Perspective (FAA)

– Methodological Perspective (Airline)

• The 3 D’s versus the 3 T’s

– Documents --Tasks

– Data --Topics

– Dialogue --Times

• Comparisons with Traditional Training


23

Three T’s of Traditional Training

• Tasks

– Flight maneuvers and procedures

– For training and checking

• Topics

– Ground training

– For training and checking

• Times

– Durations as well as intervals


24 24 24

The Three D’s of AQP

• Documents – Program Audit Data Base (PADB)

– Required AQP Documentation

• Data – Performance Proficiency Data Base (PPDB)

– De-identified student data

– Identified instructor/evaluator data

• Dialogue – Extended Review Team (ERT)

– FAA/Management/Labor Partnership


25 25

Who Uses ISD?

• Department of Defense & all branches of the

US military

– Air Force Policy Directive 36-22 mandates ISD

• Nuclear Power Plants

– Following the three mile island incident

• Petrochemical industry

• Federal Aviation Administration

– FAA Order 3140.20

• ISO Standard 10015


26 26

Instructional Systems Development (ISD) Model

• Analysis Phase

– Job Task Analysis

– Skill/Knowledge Analysis

– Student Entry Analysis

• Design Phase

– Proficiency Objectives

– Qualification Standards

– Methods & Media Selection

• Development

– Lesson Development

– Scenario Development

• Implementation Phase

– Small Group Testing

– Large Group Testing

– Instructor Training


27 27

A Systems Engineering Approach to Training Systems

• Maintenance Phase

– Training Data

– Operational/ Safety Data

– Documentation

– Methods & Media


28 28

AQP: Awesome Quantities of

Paperwork

• Phase I: Application

• AQP Application

• Phase II: Curriculum Development

• Job Task Analysis, Qualification Standards, Curriculum Outline

• I/E Job Task Analysis, Qualification Standards, Curriculum Outline

• ISD Methodology, Implementation & Operations Plan

• Phase III: Small Group Tryout

• Phase IV: Initial Operations

• Annual (review &) Report

• Maintain updates to all documents

• Phase V: Continuing Operations

• Annual (review &) Report

• Maintain updates to all documents


29 29

420 Pages of Generic Document

Templates Available from the FAA

• Application (15)

• Job Task Analysis

– Pilot (35)

– I/E (10)

• Qualification Standards

– Pilot (200)

– I/E (50)

• Curriculum Outlines

– Pilot (30)

– I/E (15)

• Instructional Systems

Development Methodology

Document (30)

• Implementation &

Operations Plan (35)


30 30

3 Basic Pilot AQP Curricula

Pilot Indoctrination

Curriculum.

IP/EP Indoc. IP/EP Qual. IP/EP Continuing Qual.

Fleet “A”Continuing Qual.

Fleet “B”Continuing Qual.

Fleet “C”Continuing Qual.

Requal.

Upgrade

Transition

Requal.

Upgrade

Transition

Requal.

Upgrade

Transition

Airbus

CL-65

F-100

AQP Overview

INDOCTRINATION QUALIFICATION CONTINUING QUAL


31 31

Evaluation: Traditional Training versus AQP

Traditional AQP

Captain F/O Captain F/O

Indoctrination

Aircraft

Systems Test

Procedures

Tng

Maneuvers

Tng

Proficiency

Check

OE

Indoctrination

Aircraft

Systems/Val

Procedures

Tng/Val

Maneuvers

Tng/Val

Loft Tng

LOE/Check

Type Rating

IOE

Line Check

Indoctrination

Aircraft

Systems/Val

Procedures

Tng/Val

Maneuvers

Tng/Val

Loft Tng,

LOE/Check

Qualification

OE

Line Check

Indoctrination

Aircraft

Systems Test

Procedures

Tng

Maneuvers

Tng

Type Rating

Check

IOE

Line Check


32 32

AQP Data Gathering Points-Pilots

AQP Continuing Qual First Look

Maneuver Validation

LOE

Annual Line Check

AQP Qualification

Systems Validation (Oral)

Procedures Validation

Maneuver Validation

LOE

IOE Line Check

Performance Proficiency

Data Base


33 33

AQP Data

• Designation • Airline, Fleet, Program,

Curriculum, Event, Date (mm/yy)

• Event • Event type, Session Grade

(pass/fail)

• Measured Item • ID, Name, item Grade (1-5) ,

Meaning, Skill/ Reason Code, Comments

• Participants • Crewmember ID, Crew ID,

Evaluator ID, Inspector ID

• Environment • Simulator ID, Seat Sub?

PF/PM, Seat, Geographic Area

• Curriculum Items • Currency item, Critical item


34 34


35 35

Sample 4 Point Grade Scale

Sample Technical Scale: Altitude

– 4 = +/- 50 Feet (Half Practical Test Standards)

– 3 = +/- 100 Feet (PTS Standard)

– 2 = Minor Deviations Quickly Corrected

– 1 = Major Deviations Slowly Corrected


36 36

Sample Grade Scale

Sample Technical Scale: Altitude

– 4 = +/- 50 Feet (Half Practical Test Standards)

– 3 = +/- 100 Feet (PTS Standard)

– 2 = Minor Deviations Quickly Corrected

• Corrective action initiated prior to exceedence

– 1 = Major Deviations Slowly Corrected

• Corrective action not initiated prior to exceedence


37 37

Sample Threat and Error Management

(TEM) Grade Scale

Sample TEM-Based Scale: All Tasks – 4 = Threats Managed

– 3 = Errors Managed

– 2 = Undesired State Managed

– 1 = Undesired State Not Managed


38 38

Sample Questions on Standard

Report 1. How many pilots/flight attendants/dispatchers and evaluations

were graded?

2. Which items are being performed for each evaluation type and how often?

3. What is the grade distribution for each evaluation?

4. Which items are receiving low or failing grades?

5. Are any currency, critical or first look items receiving low or failing grades?

6. How often are seat substitutes being used (pilots only)?

7. What is the average grade assigned by each evaluator?

8. What is the distribution of grades for each evaluator?

9. What are the most frequent skill/reason codes for low or failing grades?

10. What skill/reason codes are given for each low or failing grade?


39

Integrate the 4 Levels

Level Description Typical Measures

I Student

Reaction

Questionnaire, Interview,

Student Comments

II Student

Learning

Validation and Evaluation Gates,

Final Exam

III Learning

Transfer

Line Checks, ASAP

IV Results Incidents, Accidents, Safety

Events


40 40

Role of AQP Program Office

• Provide training and support to FAA and air carrier

personnel throughout the life of the AQP.

• Review all AQP document and data submissions.

• Issue and maintain approval letters for ERT.

• Prepare data reports for ERT.

• Support annual reviews.

• Assist/train field FAA personnel.

• Maintain national policy guidance.

• Support AFS-200 Human Factors Research

Program


41 41

12 Unique AQP Concepts

• Qualification Standards Become the New

Regulatory Compliance Document

• Crew Training & Checking is Required

• Non-Critical Tasks May be Tested Less

Frequently

• Currency Tasks May be Checked on the

Line

• Instructor/Evaluator Training is Enhanced

• Evaluators are Calibrated


42 42

12 Unique AQP Concepts

• Special Tracking Programs are Required

• Additional Validation Gates are added

• Data Collection and Reporting is Required

• Line Checks are Full Crew Checks

• Line Oriented Evaluation (LOE) is Mandatory

• All Approvals are Through a Joint FAA

Headquarters/Field Team


43

AQP Improvements in Training

• Partnership for Safety

• SMS Compliant

• Crew-Oriented Training

and Evaluation (LOE)

• Event Set Scenario

Development Methods

• Generic Documents

• Enhanced I/E Training

• Multi-point Grading

• Evaluator Calibration

Tools & Methods

• Special Tracking

• First Look (Pretests)

• Random (No Notice)

Line Check

• Extended Review Team


44 44

References

Part 121, Subpart Y (Advanced Qualification Program) Replaced SFAR-58 in 2005

Advisory Circular AC 120-54A (Advanced Qualification Program) Replaced AC 120-54 in 2006

FAA Order 8900.1 (FSIMS) (General Technical Administration, Volume 3, Chapter 21, (The Advanced Qualification Program), Sections 1-5. Replaced FAA Orders 8300, 8400 & 8700 in 2007


45 45

Questions?

• “Errare Humanum Est “To Err is Human”

-Plutarch 100 BCE

• To recover is good training