module 5 spi & alospdevelopment · nonetheless, caas does not warrant the currency, accuracy or...

10/01/2017

1

Copyright © 2015 CAAS. No part of it may be used, circulated, quoted, or reproduced for distribution outside CAAS without prior written permission. Every care has been taken to ensure that the

information contained here is accurate. Nonetheless, CAAS does not warrant the currency, accuracy or completeness of the information contained in this document, nor does CAAS accept liability

for loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from reliance on the information contained in this presentation.

Module 5SPI & ALoSP Development

Copyright © 2015 Civil Aviation Authority of Singapore. All rights reserved.

Section 1: Intermediate objectives

1. Collate potential SPIs

2. Select SPIs package

3. Develop SPI charts

4. Incorporate SPI Alert & Target settings

5. Monitor individual SPIs’ performance

6. Quantify summary SPIs performance

7. Establish and achieve ALoSP

8. SPI & ALoSP Excel Worksheet

10/01/2017

2


1. Collate potential SPIs (SMS) [Sht 2]

SPI descriptionAlert level

criteria

Target level

criteria

SPI

description

Alert level

criteria

Target level

criteriaSPI description

Alert level

criteria

Target level

criteriaSPI description

Alert level

criteria

Target level

criteria

SSP Air Operators High

Consequence SPI No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

SSP Lower

Consequence SPI

No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

Air Operator SMS High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

Air Operator SMS

Lower Consequence

SPI No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

etc. etc. etc. etc.

SSP Aerodromes High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

SSP Aerodromes

Lower

Consequence SPI

No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

Aerodrome SMS High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

Aerodrome SMS

Lower Consequence

SPI No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

etc. etc. etc. etc.

SSP ATS High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

SSP ATS Lower

Consequence SPI

No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

ATS SMS High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

ATS SMS Lower

Consequence SPI No

1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

etc. etc. etc. etc.

SSP AMOs High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

SSP AMOs Lower

Consequence SPI

No 1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

AMO SMS High


Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement between

each annual Mean

Rate.

AMO SMS Lower

Consequence SPI No

1

Ave + 1/2/3 SD.

(annual or 2

yearly reset)

__% (eg 5%)

improvement

between each

annual Mean Rate.

etc. etc. etc. etc.

Correlation of Safety Indicators (SSP) & Safety Performance Indicators (SMS)

SSP Safety Indicators (Aggregate State) SMS Safety Performance Indicators (Individual Service Provider)

High Consequence Indicators

(Occurrence/ Outcome-based)

Lower Consequence Indicators

(Event/ Activity-based)

High Consequence Indicators

(Occurrence/ Outcome-based)

Lower Consequence Indicators

(Event/ Activity-based)

Air Operators

Aerodrome Operators

ATS Operators

POA/ DOA/ MRO Organizations

Other Aviation Sectors, etc



SI Alert Level/ Criteria

(for 2010)

Alert level

Not

Breached

[Yes/ No]

SI Target level/ criteria

(for 2010)

Target

Achieved

[Yes/ No]

1

Alpha Airline’s Combined

fleet monthly serious

incident rate (e.g. per

1 000 FH)

Average + 1/2/3 SD

(annual or 2 yearly reset)Yes

5% improvement of the

2010 average rate over

the 2009 average rate

No

2

Alpha Airline’s A320 fleet

monthly serious incident

rate (e.g. per 1 000 FH)

Average + 1/2/3 SD





No

3

Alpha Airline’s A320 fleet

engine IFSD incident rate

(e.g. per 1 000 FH)

Average + 1/2/3 SD





Yes

4

Alpha Airline’s B737 fleet

monthly serious incident

rate (e.g. per 1 000 FH)

Average + 1/2/3 SD





No

5

Alpha Airline’s B737 fleet

engine IFSD incident rate

(e.g. per 1 000 FH)

Average + 1/2/3 SD





Yes

High Consequence SMS-SPIs

Safety Indicator (SI) Description

10/01/2017

3



SI Alert Level/ Criteria

(for 2010)

Alert level

Not

Breached

[Yes/ No]

SI Target level/ criteria

(for 2010)

Target

Achieved

[Yes/ No]

1

Operator combined fleet

monthly incident rate (e.g.

per 1 000 FH)

Average + 1/2/3 SD





No

2

Operator internal QMS

annual audit LEI % or

findings rate (findings per

audit)

More than 25% average

LEI or any Level 1 finding

or more than 5 Level 2

findings per audit

Yes




Yes

3

Operator voluntary hazard

report rate (e.g. per

1 000 FH)

TBD TBD

4

Operator DGR incident

report rate (e.g. per 1 000

FH)

Average + 1/2/3 SD

(annual or 2 yearly reset)No




Yes

Lower Consequence SMS-SPIs

Safety Indicator (SI) Description


3. Develop SPI charts – Data Collection & Tabulation

Mth

Alpha

Fleet Total

FH

All Alpha

Mandatory

Incidents

Incident

Rate*#

Ave

(line)

jan 51,837 10.00 0.19 0.16

feb 48,406 15.00 0.31 0.16

mar 53,354 7.00 0.13 0.16

apr 52,513 4.00 0.08 0.16

may 54,037 9.00 0.17 0.16

jun 52,673 6.00 0.11 0.16

jul 54,086 5.00 0.09 0.16

aug 54,043 13.00 0.24 0.16

sep 52,383 7.00 0.13 0.16

oct 53,042 10.00 0.19 0.16

nov 51,353 7.00 0.14 0.16

dec 53,006 9.00 0.17 0.16

Preceding Year

Mth

Alpha

Fleet Total

FH

All Alpha

Mandatory

Incidents

Incident

Rate*#

dec 53006 9.00 0.17

jan 51635 9.00 0.17

feb 44295 8 0.18

mar 48323 10 0.21

apr 47176 11 0.23

may 47469 13 0.27

jun

jul

aug

sep

oct

nov

dec

Current year

10/01/2017

4


Develop SPI charts – chart generation

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

jan feb mar apr may jun jul aug sep oct nov dec

PRECEDING YEAR ALL OPERATORS

MANDATORY INCIDENT RATE (PER 1000FH)

PRECEEDING YEAR AVERAGE (AVE)

dec jan feb mar apr may jun jul aug sep oct nov dec

CURRENT YEAR ALL OPERATORS

MANDATORY INCIDENT RATE (PER 1000 FH)

Copyright © 2015 Civil Aviation Authority of Singapore. All rights reserved. 8

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40


PRECEDING YEAR ALL OPERATORS MANDATORY INCIDENT RATE (PER

1000FH)

PRECEEDING YEAR AVERAGE (AVE)

Ave+1 SD

Ave+2 SD

Ave+3 SD

Target


CURRENT YEAR ALL OPERATORS MANDATORY INCIDENT RATE (PER 1000 FH)

CURRENT YEAR TARGET AVERAGE

A) Alert Level Setting:

Alert level for a new monitoring period (current year) is based on the

preceding period's performance (preceding year), namely its data points Average

& Std Deviation (STDEVP). The 3 alert lines are Ave+1SD, Ave+2SD and

Ave+3SD.

C) Target Level Setting (Planned Improvement) :

Target setting may be less structured than Alert level setting - eg target the new

(current year) monitoring period's Ave rate to be say 5% lower (better) than the

preceding period's Ave value.

B) Alert Level Trigger:

An Alert (abnormal/ unacceptable trend) is indicated if ANY of the conditions

below are met for the current monitoring period (current year):

- Any single point is above 3 SD line

- 2 or more consecutive points are above 2 SD line

- 3 or more consecutive points are above 1 SD line

When an Alert is triggered (potential high risk or out of control situation),

appropriate follow-up action is expected, such as further analysis to determine

source and root cause of the abnormal incident rate and any necessary action

(including HIRM) to address the unacceptable trend .

D) Target Achievement:

At end of the current year, if the Ave rate for the current year is at least 5% or

more lower than the preceding year's Ave rate, then the set Target of 5%

improvement is deemed to have been achieved.

E) Alert & Target Levels - Validity Period:

Alert & Target levels should be reviewed/reset for each new monitoring period,

based on the equivalent preceding period's Ave rate & SD, as applicable.

Alert and Target Settings

10/01/2017

5


4. Incorporate SPI Alert & Target settings

Mth

All

Operators

Total FH

All

Operators

Mandatory

Incidents

Incident

Rate*

Ave

(line) Mth

All

Operators

Total FH

All

Operators

Mandatory

Incidents

Incident

Rate*

Preceding

Year Ave

+1SD (line)

Preceding

Year Ave

+2SD

(line)

Preceding

Year Ave

+3SD

(line)

jan 51,837 10.00 0.19 0.16 dec 53006 9.00 0.17

feb 48,406 15.00 0.31 0.16 jan 51635 9.00 0.17 0.23 0.29 0.35 0.15

mar 53,354 7.00 0.13 0.16 feb 44295 8 0.18 0.23 0.29 0.35 0.15

apr 52,513 4.00 0.08 0.16 mar 48323 10 0.21 0.23 0.29 0.35 0.15

may 54,037 9.00 0.17 0.16 apr 47176 11 0.23 0.23 0.29 0.35 0.15

jun 52,673 6.00 0.11 0.16 may 47469 13 0.27 0.23 0.29 0.35 0.15

jul 54,086 5.00 0.09 0.16 jun 0.23 0.29 0.35 0.15

aug 54,043 13.00 0.24 0.16 jul 0.23 0.29 0.35 0.15

sep 52,383 7.00 0.13 0.16 aug 0.23 0.29 0.35 0.15

oct 53,042 10.00 0.19 0.16 sep 0.23 0.29 0.35 0.15

nov 51,353 7.00 0.14 0.16 oct 0.23 0.29 0.35 0.15

dec 53,006 9.00 0.17 0.16 nov 0.23 0.29 0.35 0.15

Ave 0.16 dec 0.23 0.29 0.35 0.15

SD 0.06

Ave+1SD Ave+2SD Ave+3SD * Rate Calculation:( per 1000 FH)

0.23 0.29 0.35

Current

Year

Target

(line)

Preceding Year Current year

Current Year Alert Level setting criteria

is: Preceding Year Ave + 1/2/3 SD

Current Year Target is say 5% Ave rate

improvement over the Ave rate for the

preceding year, which is:

0.15

Current Year Alert Levels


Standard Deviation - definition

� The standard deviation is a measure of how widely values are dispersed from the average value (the mean).

� A measure of the volatility of a data set.

10

10/01/2017

6


Rationale for using Ave+SD Alert setting criteria

11

0

1

2

3

4

5

6

7

8

0 1 2 3 4 5

Chart A

Incident Rate Ave Linear (Ave)

Month Incident Rate Ave

Jan 1 4

Feb 7 4

Mar 1 4

Jun 7 4

Ave 4

STDEVP 3

Month Incident Rate Ave

Jan 3 4

Feb 5 4

Mar 3 4

Jun 5 4

Ave 4

STDEVP 1

0

1

2

3

4

5

6

0 1 2 3 4 5

Chart B

Incident Rate Ave Linear (Ave)


Generate Alert and Target lines

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40


PRECEDING YEAR ALL OPERATORS

MANDATORY INCIDENT RATE (PER

1000FH)

Ave+1 SD

Ave+2 SD

Ave+3 SD

Target


CURRENT YEAR ALL OPERATORS

MANDATORY INCIDENT RATE (PER 1000

FH)

10/01/2017

7


5. Monitor individual SPIs’ performance

13

1 or more

point > 3SD

3 or more

points > 1SD

2 or more

points > 2SD


6. Quantify summary SPIs performance

SI Alert Level/

Criteria (for 2010)

Alert level

Not

Breached

[Yes (4), No

(0)]

SI Target level/ criteria (for 2010)

Target

Achieved

[Yes (3), No

(0)]

1 High Consequence

SPI No 1

Preceding Period Ave

Rate + 1/2/3 SD 45 % improvement of Current Period

Ave rate over Preceding Period Ave

rate.

0

2 High Consequence

SPI No 2




rate.

0

3 High Consequence

SPI No 3

Preceding Yr Ave



rate.

3

4 Etc

Sub-total 8 Sub-total 3

Max 12 Max 9

SI Alert Level/

Criteria (for 2010)

Alert level

Not

Breached

[Yes (2), No

(0)]


Target

Achieved

[Yes (1), No

(0)]

1 Lower Consequence

No 1Preceding Period Ave

Rate + 1/2/3 SD2

3 % improvement of Current Period


rate.

1

2 Lower Consequence


Rate + 1/2/3 SD2



rate.

0

3 Lower Consequence


Rate + 1/2/3 SD0



rate.

0

4 Lower Consequence

No 4




rate.

0

5 Lower Consequence

No 5




rate.

1

6 Etc


Max 10 Max 5

No Alert % 72.7% Target Achieved % 35.7%

58.3%

Illustration of SSP ALoS Performance Measurement

High Consequence Safety Indicators

Safety Indicator (SI)

Description

Lower Consequence & Supplementary Safety Indicators


Description

Overall SSP ALoS Performance

10/01/2017

8


7. Establish and achieve an ALoSP

SI Alert Level/

Criteria (for 2010)

Alert level

Not

Breached

[Yes (4),

No (0)]


Target

Achieved

[Yes (3),

No (0)]

1 High Consequence

SPI No 1




rate.

0

2 High Consequence

SPI No 2




rate.

0

3 High Consequence

SPI No 3

Preceding Yr Ave

Rate + 1/2/3 SD4



rate.

3

4 Etc


Max 12 Max 9

SI Alert Level/

Criteria (for 2010)

Alert level

Not

Breached

[Yes (2),

No (0)]


Target

Achieved

[Yes (1),

No (0)]

1 Lower Consequence


Rate + 1/2/3 SD2



rate.

1

2 Lower Consequence


Rate + 1/2/3 SD2



rate.

0

3 Lower Consequence


Rate + 1/2/3 SD0



rate.

0

4 Lower Consequence

No 4




rate.

0

5 Lower Consequence

No 5




rate.

1

6 Etc


Max 10 Max 5

No Alert % 72.7% Target Achieved % 35.7%

58.3%

Illustration of SSP/ SMS ALoS Performance Measurement High Consequence Safety Indicators


Description

Lower Consequence & Supplementary Safety Indicators


Description

Overall SSP/ SMS ALoS

Performance


SPI & ALoSP Excel Worksheet >>>

Quizzes >>

10/01/2017

9


SPI & ALoSP Development Worksheet – Quiz 2

During a service provider’s process of defining its potential SMS-SPIs, it should take into consideration:

a)the CAA’s SSP-SPIs for its own sector

b)the need for its SMS-SPIs to be agreed by the CAA

c) the SPIs of other equivalent service providers within its own sector

d)all above are correct



The purpose of compiling recent historical data during the development of a given SPI is to facilitate the setting of Alert and Target values which take into consideration the:

a)recent performance of the SPI

b)average occurrence rate of the recent data set

c) standard deviation of the recent data set

d)all of the above

10/01/2017

10



By the statement “Alert and Target settings will serve as quantitative performance markers for a SPI chart”, we mean to say that they can help us to interpret the performance of a SPI data set, by assessing whether there had been any abnormal occurrence rate increase, as well as whether it had achieved the planned occurrence rate reduction:

a)TRUEb)FALSE



The Alert setting of a incident rate SPI chart serves as a:

a)continuous monitor for any abnormal increase in incident rate

b)continuous monitor for any abnormal decrease in incident rate

c) one time trigger for any abnormal increase in incident rate

d)one time trigger for any abnormal decrease in incident rate

10/01/2017

11



The Target setting of an incident rate SPI chart serves as a planned improvement target for the:

a)preceding monitoring period

b)current monitoring period

c) Both (a) and (b)


SPI & ALoSP Development Worksheet – Quiz 7a

The achievement or non-achievement of a Target setting is to be established at:

a)end of the Current monitoring period only

b)every data point update interval of the Current monitoring period

c) five year intervals only

10/01/2017

12



The reason for establishing three separate Alert lines instead of a single Alert line for a SPI chart is to capture unacceptable high single data point spikes as well as sustained lower consecutive data points spikes of a SPI’s data trend:

a)TRUEb)FALSE



Target setting for individual SPI charts and Target setting for the overall summary performance of a SPIs package are one and the same thing

a)TRUEb)FALSE

10/01/2017

13



Based on the ALoSP measurement mechanism illustrated in this section, if an Accountable Executive had established the organization’s ALoSP for a given year as 70%, and the actual SPIs package summary performance came out to be 72%; this means that the organization has:

a)achieved its ALoSP for that year

b)not achieved its ALoSP for that year

c) has achieved its ALoSP for the preceding year


Congratulations

for completing this Module 5 on SPI & ALoSP Worksheet

You are now able to:

� Collate potential SPIs

� Select a package of High/ Lower consequence SPIs

� Develop SPI charts

� Incorporate SPI Alert & Target settings

� Monitor individual SPI’s performance

� Quantify summary SPIs’ performance

� Establish and achieve an ALoSP

� Use the Excel SPI & ALoSP worksheet in this module to

develop SPIs

module 5 spi & alospdevelopment · nonetheless, caas does not warrant the currency, accuracy or...

Documents