Measuring performance, benchmarking

and setting objectives

ATC Global Session 3

Ralph RIEDLE

Chairman Performance Review Commission

17 September 2014

Measuring performance, benchmarking and setting objectives 2

Topics

ANS in the European aviation context

Performance oriented approach in ANS

ANS performance review examples

Conclusions

Measuring performance, benchmarking and setting objectives 3

ANS in the European aviation context (1/2)

≈ 6% of Airline

operating costs (Europe)

Air transport delay (2013)

All ≈ 9 min. per flight

ANS-related ≈1 min. per flight

≈ 6% of aviation related CO2

emissions (0.2% of total emissions)

Safety is the primary objective! No accident with ANS contribution since 2011 Reported incidents in 0.3% of flights

Although ANS is comparatively small in aviation context….

GDP from Air Transport

in EU ≈ $160 B Source: ATAG

Air Navigation Services

≈ $10 B

Safety Cost-efficiency

CapacityEnvironment

European IFR traffic

Measuring performance, benchmarking and setting objectives 4

ANS in the European aviation context (2/2)

ANS generates….

Value….. Safety Efficient flow of air traffic

Costs …. Total economic cost ≈ $ 14 B (SES area)

Direct ANS provision costs (user charges) Indirect service quality related costs

(delays, non-optimum flight profiles) Airborne equipment costs to be added

Environmental impact

High penalties to economy if disrupted

Scope for improvements!

Total

economic

cost

≈ $14 B

Flight

efficiency

ANS related

delays

ATCO

costs

Other costs

Direct

ANS

Cost

≈ $10 B

Support

costs

CAPEX

Indirect

service

quality

related

costs

≈ $4 B

… the stakes are high!

SES area

Measuring performance, benchmarking and setting objectives 5

Performance-oriented approach in ANS (1/5)

You can’t manage what

you don’t measure!

Measuring performance, benchmarking and setting objectives 6

Performance-oriented approach in ANS (2/5)

Objectives (What should be achieved?)

Commonly accepted framework (What should be measured, targeted: KPAs, KPIs?)

Performance targets (How much should be achieved? When?)

Performance monitoring (Are we on track?)

Corrective measures (As necessary)

Strategy & objectives

(what must be done well to implement strategy )

Framework, KPIs & PIs

(How successshould be measured)

Target setting(Quantification of what should be achieved)

Reporting, monitoring& analysis

(actual vs. targets)(understand achieved

performance)

Performance Review

Verified

Data

Measuring performance, benchmarking and setting objectives 7

Performance-oriented approach in ANS (3/5)

ECAC institutional strategy (1997)

Common ATM Performance strategy Focus on outcome, not only means (e.g. technology)

Independent Performance Review Commission (PRC) (1998) Independent Performance review Light-handed regulation: objective information, recommendations

Single European Sky (SES II: 2009)

SES Performance scheme Enforceable performance targets (EU, national/FAB levels), incentives Performance Review Body (PRB) advising the European Commission

ECAC Common ATM Performance

strategy

PRC

1997 1998 2004

SES PRB

2010 2009

SES II

2012 2015

Start RP1

Start RP2

1990

Delay crisis

Delay crisis

Measuring performance, benchmarking and setting objectives 8

Performance-oriented approach in ANS (4/5)

Safety

Cost-

effectiveness

Flexibility

ICAO

high-level expectations of the ATM community

Access &

Equity

Participation

Security

Efficiency

Predictability

Interoperability

Capacity

Environment

European focus

areas for target

setting

Safety

management

Capacity

management

Cost

management

Environmental sustainability

European focus areas in line with the 11 Key performance areas (KPAs)

defined in ICAO “Global Air Traffic Management Operational Concept”

Binding targets for Safety, Cost-efficiency, Capacity and Environment

Other KPAs measured

Performance-oriented approach in ANS (5/5)

Over Time

Benchmarking dimensions

Within region

Across regions

Different dimensions of performance benchmarking enable to:

• Understand and position own performance

• Identify performance gaps and scope for improvement

• set ambitious but realistic targets

Well established publications on ANS performance in Europe:

• Annual Performance Review Report (system view, all KPAs)

• ATM Cost Effectiveness Benchmarking (ACE) reports (Economics)

• Ad hoc reports (US/ Europe comparisons, etc.)

9

European focus

areas for target

setting

Safety

management

Capacity

management

Cost

management

Environmental

sustainability

Measuring performance, benchmarking and setting objectives

Annual Performance Review Reports (PRR)

2,2

2,9

4,5

2,9

2,5

1,4

0,9

0,8 0,9 1,0 1,2 1

,4

0,9

2,0

1,1

0,6

3

0,5

3

70

80

90

100

110

120

130

140

150

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

19

97

19

98

19

99

20

00

20

01

20

02

20

03

20

04

20

05

20

06

20

07

20

08

20

09

20

10

20

11

20

12

20

13

Tra

ffic

in

de

x (

ba

se

: 1

99

7)

En

-ro

ute

AT

FM

de

lay/ flig

ht (m

in.)

ATC Other (strike, equipment, etc.) WEATHER

OTHER (Special event, military, etc.) IFR Traffic

Average en-route ATFM delay per flight

source: Network Manager

4,9

1

4,8

7

4,8

6

3,3

1

3,2

0

3,1

4

2,0

2,5

3,0

3,5

4,0

4,5

5,0

5,5

6,0

200

9

2010

2011

2012

2013

Flight Plan (KEP) Actual trajectory (KEA)PRU analysis

ineffic

ien

cy

(%)

2009/10 KEA data based on regression analysis

• Verified facts and performance indicators

• Independent critical analysis (wide spectrum)

• Recommendations to decisions makers

SAFETY: Review of safety incident evolution

in order to identify trends (RI, SMI, etc.)

COST-EFFICIENCY: Analysis of ANS unit costs

und underlying drivers

ENVIRONMENT: Monitoring of flight

efficiency en route and in terminal areas;

CAPACITY: Evaluation of ANS related en route

and airport delays and constraining factors;

Safety Cost-efficiency

CapacityEnvironment

European IFR traffic

Measuring performance, benchmarking and setting objectives 10

ANSP benchmarking – ACE reports

ATM Cost Effectiveness Benchmarking (ACE) reports

Widely accepted performance framework (ICAO Doc 9161)

Verified data, official performance indicators and metrics

Factual analysis at European level, within region, over time

Purpose: Inform decision makers, support target setting within SES, etc

Safety Cost-efficiency

CapacityEnvironment

European IFR traffic

11 Measuring performance, benchmarking and setting objectives

Global perspective: US-Europe comparison

2012/2013 Europe USA US vs.

Europe Geographic Area (million km

2) 11.5 10.4 ≈ -10%

Nr. of civil en route Air Navigation Service Providers 37 1

Number of en route centres 63 20 -43

Number of APP units (Europe) and terminal facilities (US) 260 163 -97

Number of fully trained Air Traffic Controllers (ATCOs in Ops.) 17 200 13 400 ≈ -22%

Total staff 58 000 35 500 ≈ -39%

Controlled flights (IFR) (million) 9.6 15.1 ≈ +57%

Flight hours controlled (million) 14.3 22.4 ≈ +57%

Average length of flight (within respective airspace) 551 NM 515 NM ≈ -7%

Relative density (flight hours per km2) 1.2 2.2 ≈ x1.7

Share of General Aviation 3.9% 21%

Facts

Similar conditions (geographical area, average flight length)

US controls more IFR traffic (+57%) with fewer staff (-39%),

including ATC Controllers (-22%)

What are the underlying drivers of performance?

12 Measuring performance, benchmarking and setting objectives

Reactive policy in the 90’s: delays go up while costs go down, and vice versa

As of 1998, performance-oriented approach and improved capacity management contributed to reduction of both delays and unit costs

Enforceable SES performance targets apply from 2012 onwards

Performance in capacity and cost-efficiency

Traffic index

En route

delay

13 Measuring performance, benchmarking and setting objectives

5.42 5.385.18 5.15 5.11

3.29 3.17 3.12

0.0

1.0

2.0

3.0

4.0

5.0

6.0

2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019

rou

te e

xte

nsi

on

(%

)

Source: PRU analysis

Target RP1 Targets RP2

Environmental performance

Filed flight plan

Flight efficiency first measured in 2006: continuous improvement since then Good routing efficiency of ANS (≈3%) compared to other transport modes

• Yet significant economic impact (fuel burn, flight time) • Impossible to reach 0% with full civil-military traffic load

SES targets on Environment set for 2014 (FPL), 2019 (Actual, FPL)

Improved flight-efficiency compensates for air traffic growth

Carbon-neutral growth of aviation (due in 2020) already being met as far as European ANS is concerned!

Actual flown

trajectory

ENVIRONMENT:

Performance targets on

en route flight efficiency

within SES scheme

14 Measuring performance, benchmarking and setting objectives

Conclusions

Performance is the bottom line for ANS policy – Impact on more than $14B p.a. in EU, while ensuring safety

European Performance-driven strategy delivers, but margins for significant further improvements remain – Independent Performance review since 1998 (PRC)

– Stronger regulation under SES II (enforceable targets, regulations)

– EUROCONTROL supports both PRC and EC/PRB

Global benchmarking reveals best practices, weaknesses

– EUROCONTROL willing to engage with benchmarking partners

More details in workshop “Driving excellence in ATM performance” ( Workshop theatre, 19 Sep. , 10am)

Reports available at http://www.eurocontrol.int/prc/publications

15 Measuring performance, benchmarking and setting objectives