measuring safety performance - an analyst’s perspective

Measuring Safety PerformanceAn Analyst’s Perspective

By: Indunil Weerasinghe28th September, 2013

www.linkedin.com/in/indunilmwwww.slideshare.net/walk_the_safety_talk

Measuring Safety PerformanceAn Analyst’s Perspective 2

What are we trying to achieve?

• In Australia:– Civil Aviation Order 82.5 lists AOC

holder’s obligations for HCRPT *:

* HCRPT = High Capacity Regular Public Transport


• Internationally:– ICAO framework– Safety Management Manual (SMM) Doc 9859



• A key element of an effective SMS is to measure safety performance

• Collect, analyse and report safety data

• Objective: to facilitate safety assurance


Process

•Ensure data quality / integrity

•Extract reactive (lagging) indicators

•Extract proactive (leading) indicators

•Develop predictive analyses


Quality of data

• Data should be:– Valid, complete, consistent, and accurate

• Will always need some clean up, but...• Not feasible to clean it to perfection!


Quality of dataBe pragmatic:

Be mindful of the limitations of your data,but make the most of what you have available!


Three Levels of Analysis

3.Predictive

2. Proactive

1. Reactive


Three Levels of Analysis

• To demonstrate:

– A safety data set was analysed

– Not true operational data (“mock” data used for presentation purposes only)


REACTIVE INDICATORSBasic analyses to support reporting and reactive management


What are “Reactive” Indicators

• Also referred to as “Lagging” Indicators

• Basic measurements that will sum up the recent past

• Can be expressed in terms of simple graphs and charts

• For example....


Examples of Reactive Indicators

Mar-12 Apr-12 May-12 Jun-12 Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-130

10

20

30

40

50

60

70

80

90

Incidents (per month) - Whole Fleet

Fleet TotalB777A380E190

?RATE

orQTY

?


Examples of Reactive Indicators• Measuring RATE:

– E.g. Incidents per 1000FH– Communicates a relative measure of risk– Allows for comparison

• E.g. between a/c types, or across timelines

• Measuring QUANTITY:– Communicates an absolute measure of risk– Provides an understanding of the operational impact

• E.g. To determine resources required for incident investigation



35.3%

29.3%

13.2%

6.4%

5.8%2.4%

2.2% 1.6% 1.0% 2.7%

Incidents by Incident Type

Wildlife

Weather

ATS Deficiency

Laser Light

Aerodrome Related

Other Aircraft Failure to Comply

Forecast Deficiency

Other Operating Environment

Wake Turbulence

Other



Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Apr-11 May-11 Jun-11 Jul-11 Aug-11 Sep-11 Oct-11 Nov-11 Dec-110

10

20

30

40

50

60

Seasonal Variation of Weather Related Incidents

Drops off towards the end of the year (Oct/Nov/Dec)


PROACTIVE INDICATORSBasic statistical analyses to support proactive management


What are “Proactive” Indicators• Classified as a “Leading” Indicator

• Forecasting the near future based on trends

• Aims to flag problem areas before they manifest

• From ICAO’s SMM (Rev 3):– Uses “standard deviation” to set up “alert levels”


Setting the Alert Levels• Calculate monthly mean

• Let’s call this μ

• Calculate the std dev. over the same period

• Let’s call this σ

• Calculations could be done using 12 or 24 month data


When are alerts triggered?

• Reading above (μ + 3σ) in any month

• Reading above (μ + 2σ) in two consecutive months

• Reading above (μ + σ) in three consecutive months


When are alerts triggered?

Jul-12 Aug-12 Sep-12 Oct-12 Nov-12 Dec-12 Jan-13 Feb-13 Mar-13 Apr-13 May-13 Jun-1315

17

19

21

23

25

27

29

31

33

35

Incidents Last YE's μ μ + σ μ + 2σ μ + 3σ


PREDICTIVE INDICATORSAdvanced statistical analysis to support predictive management


What are “Predictive” Indicators

• Also classified as a “Leading” Indicator• Less guidance available• Calls for a more creative approach to the

analysis

• This is where understanding your data and knowing how to make the most of it will really pay off!


An example of a Predictive Indicator

• The following statistical test and its application won’t necessarily lie true of all data sets

• It’s just an example to get you thinking about how you could further analyse your data


The Chi square test

• Null hypothesis (H0):– Aircraft type category are

INDEPENDENT variables

• Chi square test on H0 yielded a P score of 0%

• Conclusion - Reject the null:• There is a highly, statistically significant

dependency


Taking away the “Stats Speak”

• A given aircraft type is more vulnerable to incidents of a certain nature

• Opportunity to predict and investigate problem areas regardless of “alert levels”:

– For example:• Risk profile to show high risk areas


What do we mean by a “Risk profile”?

• What if we were to compare...

Actual frequency of a particular type of

incident on a particular aircraft

Theoretical frequency if the incident were to happen independently

of aircraft type


A380 Risk profile “hot spots”

• This tells us “VHF Comms” incidents are 7x more likely on this a/c in my fleet than if a/c and incident types were independent

• Flags up a high risk area, regardless of an alert level being breached

• Warrants a root-cause analysis:– Is there an equipment reliability issue? Pilot training? Procedural issue?


IN CONCLUSION...


Keep in mind....All these examples are simply that... Just fake data designed to convey an

idea.

Don’t apply the conclusion!Don’t even apply the method!

Just apply the thinking!Measuring Safety Performance

An Analyst’s Perspective 29

OpportunitiesChallenges

1. Minimal guidance on “predictive indicators”

As SMS’s mature and enter latter phases of implementation, this is an area that will benefit from increasing industry & academic collaboration.

Potential for significant value addition to the “safety assurance” process


Opportunities

2. Operational safety data tells you whether you are achieving the desired safety outcomes, but...

... do you know if those outcomes are being achieved in the most efficient, least costly manner?


Opportunities A “balanced scorecard” approach to safety

metrics would complement the existing leading/lagging indicators

Better return on your safety investment

Incentivise the desired safety behaviours

Nurture a generative safety attitudes

Cultivate a continuous improvement culture


• Thanks for reading, and remember...

• The aggregate safety of our industry lies in our hands, so...

• ... the more we innovate and share our ideas, the better the outcome we can achieve.

• Please feel free to contact me:– On LinkedIn at www.linkedin.com/in/indunilmw– Or email me at [email protected]


http://www.linkedin.com/in/indunilmw

mailto:[email protected]

34

Download these Resources• ICAO’s Safety Management Manual (Doc 9859 Rev 3)

• Robert S Kaplan’s paper from the Harvard Business School on the Balanced Scoredcard

• Testing for Independence using the “Chi Square” Test, from Berkeley University

• Australian Legislative Instrument: Civil Aviation Order 82.5

Measuring Safety PerformanceAn Analyst’s Perspective

http://www.icao.int/safety/SafetyManagement/Documents/Doc.9859.3rd%20Edition.alltext.en.pdf

http://www.hbs.edu/faculty/Publication%20Files/10-074.pdf

http://www.hbs.edu/faculty/Publication%20Files/10-074.pdf

http://www.stat.berkeley.edu/users/rice/Stat2/Chapt28.pdf

http://www.stat.berkeley.edu/users/rice/Stat2/Chapt28.pdf

http://www.comlaw.gov.au/Details/F2012C00213/797dc9a6-e5b3-4fd4-a1ae-0b329d96ea7d

measuring safety performance - an analyst’s perspective

Technology