feast chetcuti

39
The FEAST Service - Combining the professional with the practical across organizational boundaries Fidel Chetcuti Fidel Chetcuti DAP/SSH DAP/SSH EUROCONTROL EUROCONTROL FEAST Service FEAST Service

Upload: splycho-goran

Post on 15-Dec-2015

38 views

Category:

Documents


8 download

DESCRIPTION

...

TRANSCRIPT

Page 1: FEAST Chetcuti

1 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

The FEAST Service - Combining the professional with the practical

across organizational boundaries

Fidel ChetcutiFidel ChetcutiDAP/SSHDAP/SSH

EUROCONTROLEUROCONTROL

FEAST ServiceFEAST Service

Page 2: FEAST Chetcuti

2 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Overview

Background and development

Initial Validation Study

Using FEAST for decision making

Future Challenges

Predictive Validation Study

Current FEAST Package

Making FEAST work in practice

Questions and Discussion

Page 3: FEAST Chetcuti

3 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

F

E

S

A

T

IRST

UROPEAN

IR TRAFFIC CONTROLLER

ELECTION

EST

Page 4: FEAST Chetcuti

4 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST ServiceFEAST Service

4

The FEAST Service Team

FEAST Service Expert

[email protected] Service Manager

[email protected] Service Assistant

[email protected]

Marc Damitz Fidel Chetcuti Michaela Heese

Page 5: FEAST Chetcuti

5 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

KEY FEATURES

Professional standards crossing national boundariesProfessional standards crossing national boundaries

+

Web platform quick, cost effective and easy to useWeb platform quick, cost effective and easy to use

+

State of the art testing option conforming to EUROCONTROL Selection Guidelines. State of the art testing option conforming to EUROCONTROL Selection Guidelines.

Standardised administrative procedures Europe-wide. Fairly simple to use and manageStandardised administrative procedures Europe-wide. Fairly simple to use and manage

+

FEAST User Group and CommunityFEAST User Group and Community+

+

Page 6: FEAST Chetcuti

6 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

KEY I.T. ISSUES

QUALITY AND STANDARDSSetting and maintaining European standards

QUALITY AND STANDARDSSetting and maintaining European standards

SECURITY AND CONFIDENTIALITYSECURITY AND CONFIDENTIALITY

DATA scores and material for research and

validation

DATA scores and material for research and

validation

REAL TIME AND ONLINEREAL TIME AND ONLINE

Page 7: FEAST Chetcuti

7 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Job and Task

analysis

Model of knowledge,

skills, abilities and other

characteristics

Find tests to

measure KASOs

FEAST test

package

THE BASIS OF FEAST

Page 8: FEAST Chetcuti

8 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

TASK ANALYSIS

Behaviour

Decision making

competence

Skills and abilities

Knowledge

Situational Situational AwarenessAwareness

Page 9: FEAST Chetcuti

9 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

THE FEAST MODEL of 26 KASOs

English asthe working

language

&

Computerskills

PerceptionComprehension

Projection(Planning &

Problem Solving)

Decision-making

Situational Awareness

Personality

Characteristics

(Behaviouralelements)

(Skills, Abilities)

(Knowledge)

Page 10: FEAST Chetcuti

10 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

CONTROLLER SKILLS AND ABILITIES

Perception of environmentPerceptual SpeedSelective AttentionMedium-term memoryMultiple-task performanceScanning abilityMonitoring/Vigilance

Comprehension of current

situation3-dimensional abilityVerbal comprehensionConscientiousnessMental Flexibility

Projection of future statusPlanning ability

Problem solving

Decision-makingDecision-makingCommunicationDecisivenessComposure/ResilienceAdaptability

Page 11: FEAST Chetcuti

11 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Personality related and behaviouralcompetences

PERSONALITY AND LEARNED BEHAVIOUR

Composure/resilience

Self-awareness

Achievement motivation Flexibility

Social sensitivity

Emotional stability

Decisiveness

Co-operation

Conscientiousness

Page 12: FEAST Chetcuti

12 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

GETTING THE RIGHT TESTS AND GETTING THE TESTS RIGHT!

Available tests

Individual tests evaluated against minimum reliability and validity requirements

Further ratings

according to 13 other criteria

Detailed questions

to shortlisted

test suppliers

FEAST test

battery

Page 13: FEAST Chetcuti

13 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

CONCURRENT VALIDATION STUDY

3 populations

(ATC trainees,ATCOs and unselected applicants)

10 volunteer ANSPs

577 test takers

FEAST web•low reliability for 1 test , other tests satisfactory•Low validity between single tests and BOS overall score, but significant correlation between BOS criteria and composite test scores

FEAST SDM•Reliability not rechecked but test developer reported satisfactory•Low validity between overall score and BOS overall but significant correlation between BOS criteria and various raw scores and composites in both groups

Conclusions: despite the issues encountered (sample size, care with BOS completion, motivation of

participants), results sufficiently encouraging to start to use FEAST carefully

Page 14: FEAST Chetcuti

14 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST CURRENT VERSION

FEAST TEST BATTERY

COGNITIVE ABILITY Heading and Range (SAHR)

English Listening and Comprehension (ENA)Planning (SAP)Attention (SORT)- Multi-tasking (PMT) andVisualization (FOLD)

KNOWLEDGE English language listening and

comprehension

WORK SAMPLE

Strip Display Management

INTERVIEWSituational Interview

Page 15: FEAST Chetcuti

15 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

What’s in FEAST?

web based run from PC 5 ability tests and 1 English

language listening comprehension test-

1 work sample test run from CD-ROM

2.5hrs 2.25hrs

Page 16: FEAST Chetcuti

16 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Selection criteria and hurdles

Cognitive abilities

Job relevant

Knowledgee.g. English language

More complex

skills/abilities e.g. multi-

tasking

Behavioural/ personality

related factors e.g.

Questionnaire, self

assessment

Personal interaction and mutual “checking”e.g. final interview

Medical exam

PHASE1 PHASE2 PHASE3FINAL

DECISION

FEAST Web FEAST SDM Assessment Centre, Interview

Selection Board

Assessment Centre

Page 17: FEAST Chetcuti

17 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Integration of FEAST Web-based Tests

1. Are proposed as Phase 1 tests to filter candidates for a second selection Phase

2. The Multiple Stage Selection Approach is recommended (‘multiple hurdle’)

FEASTweb

FEAST SDM

Inter-view

Phase 1 Decision Phase 2 Decision Phase 3 Final Decision

Page 18: FEAST Chetcuti

18 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Tests with low and high validity

True positive

False positive

False negative

True negative

Test with low validity Test with high validity

Page 19: FEAST Chetcuti

19 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Influence of the ‘base rate’

Cut-off Score

Cut-off Score

Base rate : how many people could do this job?

Lower base rate Higher base rate

Page 20: FEAST Chetcuti

20 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Influence of the ‘selection ratio’

Cut-off Score

Cut-off Score

Higher selection ratio Lower selection ratio

Selection ratio: how strict we are with pass scores

Page 21: FEAST Chetcuti

21 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST Decision Strategies

MULTI STAGE SELECTION using a combination of both the Composite & Multiple cut-off Models

Stage 1: Selection of the ‘right profile’following the Multiple cut-off Model (cut-off

score for each test, select across all tests, the Mean (Stanine 5) is used as the cut-off point)

Stage 2: Selection ‘from the top’ based on the Composite Model according to number required

Page 22: FEAST Chetcuti

22 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

TEST RESULTS AND NORMS

Results expressed as raw scores (e.g. 79 out of 120 items answered

correctly)Raw scores compared to norm group

Norms based on multinational and multicultural set of applicants Result given as stanine scores (1 to 9)Norms updated from time to time. Possibility of developing country norms in future

Individual and Group Selection Reports

Page 23: FEAST Chetcuti

23 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Composition of FEAST Overall Score

VisualisationFOLD (3DC)

Multi Tasking - AlertnessPMT

Sorting AbilitySORT 6

Mean Reaction Time (MRT)SORT 3

Planning ATCSAP

English proficiencyENA

Heading and RangeSAHR

FEAST Overall ScoreTotal number of indexes: 6

Test

OVERALL SCORE•equal weighting for each test index•Sum of all weighted indexes standardised as 0-100•Reflects pure cognitive abilities dimension•English language score is separated

Page 24: FEAST Chetcuti

24 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Decision Strategies - Example

Candidate A:

4ENA

5PMT

6SAHR

6SAP

7FOLD

8SORT

7FEAST Overall

StanineTest Result

9ENA

5PMT

6SAHR

4SAP

3FOLD

6SORT

5FEAST Overall

StanineTest Result

Candidate B:

Page 25: FEAST Chetcuti

25 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Example for combination of multiple cut-off and selection from the top

64254332Cand. F

65555654Cand. D

Selection

6

4

5

ENA

5

6

6

FOLD

5

4

7

PMT

66565Cand. C

84687Cand. B

58795Cand. A

SAHRSAPSORTFEAST overall

candidate

Stage 1: Selection of the ‘right profile’ following the Multiple cut-off Model, i.e. FEAST cognitive tests ≥ 5, ENA ≥ 4.

Stage 2: Selection ‘from the top’ based on the Composite Model, e.g. selection of the best 2 candidates.

Page 26: FEAST Chetcuti

26 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

PREDICTIVE VALIDATION

Behaviour Observation Scale BOS

Common criterion for all

countries. Available in

different languages

•Cognitive aspects ATC •Personal/inter-personal aspects of ATC job•Self confidence•English language proficiency and ATC phraseology

Common training for assessors before

completing BOS form

Relates to the same job and task analysis as the FEAST tests

Trainee assessment done

at two points during training

Page 27: FEAST Chetcuti

27 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

The BOS Form elements

Behavioural performance measurement

6 point scale coveringInferior/Comparable/Superior

Anchored Examples of inferior / superior behaviours

38 items plus General Assessment

BOS Form

Page 28: FEAST Chetcuti

28 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Validation Process

Selection tests

Final Decision

BOS Assessments

Compare BOS assessments with FEAST results

Quantify predictive power of FEAST tests

Enhance and refine

test battery

Establish and maintain close links

with trainers

Page 29: FEAST Chetcuti

29 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST Business ApplicationFEAST Business Application

Central maintenance and technical problem resolution

Hosted confidentially and securely

Database Management of job applicants

FEAST computer-based tests

Scores computed and stored for useWeb based

tools

Page 30: FEAST Chetcuti

30 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

WHAT IS IMPORTANT FOR OUR USERS

high quality

cost

effective

common

platform

perceived

fairness

harmonised process

Page 31: FEAST Chetcuti

31 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

KEY BENEFITS FOR USERS

Security

Quality and professional Standards

Validation data

FEAST User

Group

Cost effective, shared

developments

Page 32: FEAST Chetcuti

32 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

How does the FEAST Service operate?

TECHNICAL PROVIDERSInfrastructure -MISTest platform-External providerPotential new test providers

FEAST SERVICESelection testsTechnical supportProfessional supportUser Group facilitationValidation and Development

Means of delivery:WebCD_ROMHelpdeskPersonal contactFUG

USER ACTIVITIESTesting candidates

Selection

High quality process

Cost effective process

Networking opportunitiesContract and SLA

Page 33: FEAST Chetcuti

33 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST Service functions

FEAST Service Team

Implementation support and

training

User Group facilitation

Daily Helpdesk

Validation and Studies

Test Maintenance and Development

Management and co-ordination of internal

and external technical partners

Tool and methodology for validation (Behaviour Observation Scale BOS)

Page 34: FEAST Chetcuti

34 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST User Group

Facilitated by EUROCONTROL -

meets at least once a year

Forum for exchange of information and opinions

Knowledge base to enhance recruitment

and selection procedures and

standardsContinuing improvements and evolution of the

FEAST tools

A common goal:

a practical and high quality tool for selection of ab initio trainee

controllers

Page 35: FEAST Chetcuti

35 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

1619 FEAST Test Takersin the first half of 2007 (15 Users)

FEAST completed 2004-June 2007

157232 293 313

508

285 300

790

422

651

1003

1209

780 839

0 00

200

400

600

800

1000

1200

1400Ja

n-M

ar

Apr-

Jun

Jul-S

ept

Oct

-Dec

Jan-

Mar

Apr-

Jun

Jul-S

ept

Oct

-Dec

Jan-

Mar

Apr-

Jun

Jul-S

ept

Oct

-Dec

Jan-

Mar

Apr-

Jun

Jul-S

ept

Oct

-Dec

2004 (∑=995) 2005 (∑=1883) 2006 (∑=3285) 2007 (∑=1619)

time (in months)

Num

ber o

f app

lican

ts w

ho

com

plet

ed F

EAST

Page 36: FEAST Chetcuti

36 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FEAST implementation

FEAST IMPLEMENTATION CHALLENGES

Facilities, equipment, Security, IT

Roles and Responsibilities

Selection process

Validation

Training and competencies of staff

Review existing policies

Organizational issues

Identify new issues and questions

Page 37: FEAST Chetcuti

37 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

FUTURE CHALLENGES FOR ATC SELECTION

Harmonized and Harmonized and high standard high standard ATC selection ATC selection

processes across processes across EuropeEurope

Providing service to more customers while maintaining

high quality

Staying cost effective

The impact if any of cultural

differences

Spoken English language test for

selection

Behaviouralassessment

Scope for further harmonization

•Age issues•Requirements for educational level

Predictive Validation

Study

Page 38: FEAST Chetcuti

38 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007

Knowledge skills ability requirements for future ATC job

Controller job largely unchanged

Controller more passive. Skills used more infrequently but when called for,some skills

could be more demanding e.g. problem solving

Future changes to controller task likely from•Continuing automation•Standardisation and harmonisation of activities across national boundaries•More flexible use of airspace (Single European Sky, Functional Airspace Blocks) to increase capacity•Delegation of responsibilities from ground to pilot

Page 39: FEAST Chetcuti

39 FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2FEAST PRESENTATION SUMMER SCHOOL GRAZ 3 JULY 2007007 © EUROCONTROL 2007© EUROCONTROL 2007FEAST PRESENTATION FEAST PRESENTATION Brussels, 21 March 2007Brussels, 21 March 200739

FEAST ServiceFEAST Service

Don’t hesitate to ask easy questions!