kpi deployment guide · kpi key performance indicator. mission statement a brief statement of the...

EUROPEAN ORGANISATION FOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL

EUROPEAN AIR TRAFFIC MANAGEMENT PROGRAMME

KPI Deployment Guide

AIM/AEP/S-LEV/0007

Edition : 0.2 Edition Date : 19 Apr 2002 Status : Draft Class : General Public

DOCUMENT IDENTIFICATION SHEET

DOCUMENT DESCRIPTION

Document Title KPI Deployment Guide

EWP DELIVERABLE REFERENCE NUMBER

PROGRAMME REFERENCE INDEX EDITION : 0.2

AIM/AEP/S-LEV/0007 EDITION DATE : 19 Apr 2002 Abstract

This guide presents main activities required to deploy a performance measurement system.

Keywords KPI Indicator Activity Template Checklist Deployment Guide

CONTACT PERSON : Paul Bosman TEL : 3333 UNIT : AIM

DOCUMENT STATUS AND TYPE

STATUS CATEGORY CLASSIFICATION Working Draft Executive Task General Public Draft Specialist Task EATMP Proposed Issue Lower Layer Task Restricted Released Issue INTERNAL REFERENCE NAME : AHEAD Electronic Filing System

AIM/AEP/S-LEV/0007 KPI Deployment Guide

Page iv Draft Edition: 0.2

DOCUMENT APPROVAL

The following table identifies all management authorities that have successively approved the present issue of this document.

AUTHORITY NAME AND SIGNATURE DATE

Author/Editor Ertan Ozkan 19 Apr 2002

Programme Manager

Conrad Cleasby 19 Apr 2002

Quality Assurance Paul Bosman 19 Apr 2002

KPI Deployment Guide AIM/AEP/S-LEV/0007

Edition: 0.2 Draft Page v

DOCUMENT CHANGE RECORD

The following table records the complete history of the successive editions of the present document.

EDITION DATE REASON FOR CHANGE SECTIONS

PAGES AFFECTED

0.1 12 Nov 2001 Creation All

0.2 19 Apr 2002 • A. Zarbov’s comments are handled. • Section Glossary is added. • New activities are added. • Some activities are refined. • Templates and checklists are refined (new

look and feel: black & white). • Improvement Matrix is removed from

Appendix E.

All


Page vi Draft Edition: 0.2

- Page left intentionally blank -


Edition: 0.2 Draft Page vii

TABLE OF CONTENTS

1. INTRODUCTION ......................................................................................................1 1.1 Purpose and scope.................................................................................................1 1.2 References ..............................................................................................................1 1.3 Glossary ..................................................................................................................3

2. IMPROVEMENT MODEL .........................................................................................5 3. ACTIVITIES..............................................................................................................6

3.1 Activity: Prepare Prerequisites..............................................................................6 3.1.1 Activity: Identify Customer Requirements....................................................7 3.1.2 Activity: Develop Organisation Mission, Vision and Strategy .......................8 3.1.3 Activity: Identify Core Processes .................................................................9 3.1.4 Activity: Identify Process ...........................................................................10

3.2 KPI Definition ........................................................................................................11 3.2.1 Activity: Define KPIs..................................................................................11 3.2.2 Activity : Develop Data Collection Plan......................................................12

3.3 Monitoring and Analysing Data ...........................................................................13 3.3.1 Activity: Monitor KPI Compliance ..............................................................13 3.3.2 Activity: Gather and Analyse Data.............................................................14

3.4 Activity: Improve Component ..............................................................................15 3.5 Activity: Refine KPIs.............................................................................................17

4. CONCLUSIONS AND FUTURE WORK .................................................................18 APPENDIX A KPI DESIGN FORM..................................................................................20 APPENDIX B MAJOR PITFALLS ...................................................................................21 APPENDIX C KPI CHECKLIST.......................................................................................22 APPENDIX D CUSTOMER FORMS................................................................................25

D.1 Who Are Our Customers? ...................................................................................25 D.2 Customer Segments.............................................................................................27 D.3 Customer Evaluation Form..................................................................................28 D.4 Customer Background Information ....................................................................29

APPENDIX E IMPROVEMENT FORMS..........................................................................30 E.1 Improvement Matrix ................................................... Error! Bookmark not defined. E.2 Improvement Plan ................................................................................................30 E.3 Improvement Selection Checklist .......................................................................32

APPENDIX F DATA COLLECTION PLAN .....................................................................33


Edition: 0.2 Draft Page 1

1. INTRODUCTION

1.1 Purpose and scope

This document presents the general framework for the deployment of Key Performance Indicators (KPIs) within the context of a quality management system.

It contains activities derived from a simple 3-tier improvement model. Activities are given in a step-by-step manner in order to give a clear picture what is required. Each activity is given in the following template:

Purpose: Purpose of the activity Steps: Steps needed to carry out the activity Input Artefacts: Required Inputs

Resulting Artefacts: Created outputs

Frequency: Frequency of the activity Role: Who is responsible for this activity Tips and Tricks: Additional comments, tips and tricks

Each activity contains references to the applicable guides and required documents either in its input artefacts and/or its steps. If necessary, the activity is supported by a flowchart. Appendices contain some useful information like design forms, templates and checklists.

The document is organised as follows:

• Chapter 2 presents 3-tier improvement model and basic loop in a performance measurement system.

• Chapter 3 gives activities required to deploy a performance measurement system.

• Chapter 4 presents conclusions and future work

Throughout this document the term “performance measurement system” is used to denote the decision support system that contains a set of Key Performance Indicators (KPIs) with well-defined measurement, monitoring, analysis and improvement procedures.

1.2 References

[1] Introduction to KPI Models, Edition 0.1, AIM/AEP/S-LEV/00004, 29 Oct 2001.

[2] KPI Portfolio, Edition 0.4, AIM/AEP/S-LEV/0005, 23 Oct 2001.

[3] KPI Top-12, Edition 0.1, AIM/AEP/S-LEV/0006, 13 Nov 2001.


Page 2 Draft Edition: 0.2

[4] KPI Measurement, Monitoring and Analysis Guide, Edition 0.1, AIM/AEP/S-LEV/0008, 30 Nov 2001.

[5] Change Management the 5-step Action Kit, C. Rye, ISBN 0749433809, Kogan Page Limited, 2001.

[6] Major Pitfalls of Performance Measurement Systems - Module V, Randy R. LaBarge, Pacific Northwest National Laboratory, U.S. Department of Energy, PBM-SIG Fall ‘99 Meeting, 17 November 1999, Washington D.C.

[7] Handbook for Basic Process Improvement, May 1996, http://www.balancedscorecard.org/files/bpihndbk.pdf.

[8] The Process Improvement Notebook, Department of the Navy, Total Quality Leadership Office, Publication No: 97-01, 1997.

[9] Operational Performance Measurement Increasing Total Productivity, W. Kaydos, ISBN 1574440993, CRC Press LLC, 1999.

[10] The Six Sigma Way, P. S. Sande, R. P. Neuman, R. R. Cavanagh, ISBN 0-07-135806-4, McGraw-Hill, 2000.



1.3 Glossary Term Description Activity One specific action or operation composed of tasks, which are a

factor in a process. A series of activities comprise a process. Baseline A level of performance that is considered normal or average. It is

the level against which all future measurements will be compared to identify trends in performance levels.

Block Diagram A specialised, high-level type of flowchart. Its highly structured form presents a quick overview of major process steps and key process participants, as well as the relationships and interfaces involved.

Brainstorming A powerful, versatile and simple technique for generating large numbers of ideas around a common theme from a group of people in a very short period of time.

Cause A proven reason for the existence of a problem - not to be confused with symptoms.

Cause and Effect Diagram

An analysis tool to display possible causes of a specific problem or condition.

Control Chart A display of data in the order that they occur with statistically determined upper and lower limits of expected common cause variation. It is used to indicate special causes of process variation, to monitor a process for maintenance, and to determine if process changes have had the desired effect.

Corrective Action

Action(s) designed to identify and eliminate root causes of non-conformances and non-conformities.

Customer The receiver of an output of a process, either internal or external to an organisation or corporate unit. A customer could be a person, a department, a company, etc.

Effect An observable action or evidence of a problem. Flowchart A graphical representation of the flow of a process. A useful way

to examine how various steps in a process relate to each other, to define the boundaries of the process, to identify customer/supplier relationships in a process, to verify or form the appropriate team, to create common understanding of the process flow, to determine the current "best method" of performing the process and to identify redundancy, unnecessary complexity and inefficiency in a process.

Interrelations Digraph

A graphical representation of all the factors in a complicated problem, system or situation.

KPI Key Performance Indicator. Mission statement

A brief statement of the processes or products a work unit is expected to perform as part of a larger work unit, or for its customers.

(KPI) Objective Target performance level set for the KPI Pairwise Ranking

A structured method for ranking a small list of items in priority order.

Post-It Small piece of (yellow) sticky paper used to take notes. Prioritisation Process of ranking ideas, projects or solutions as determined by

individual judgment and group consensus Process Combination of people, machines, tools, techniques and

materials linked in a systematic series of steps or actions to complete a task or fulfil a commitment.

Quality An organised way of documenting the improvement process of a



Term Description Assurance Storytelling

team that is working systematically to resolve a specific problem and/or improve a given process.

Requirement Valued product or service features such as safety, timeliness, accuracy, completeness, cost, dimension and quantity.

Root Cause The basic reason creating an undesired condition or problem. In many cases, the root cause may consist of several smaller causes.

Root Cause Analysis

Using one or more various tools to determine the root cause of a specific failure.

SIPOC Map A tool used to identify all relevant elements (i.e., supplier, input, process, output and customer) of a process within the context of process management and improvement.

Stakeholder Anyone for whom an organisation or an individual within an organisation provides goods or services.

Strategy Activity that leads to achieving the vision. System Modelling

Shows how the system should be working by using three elements: inputs, processes and outcomes.

Vision statement A statement of a shared dream for a community or group. Voice of the Customer

A disciplined, cyclical approach to obtaining, understanding and prioritising customer wants and needs (requirements).



2. IMPROVEMENT MODEL

The main objective of a performance measurement system is to provide a basis (i.e., facts) for continual improvement:

Figure 1. 3-Tier Improvement Model

The above improvement model summarises basic skills required to deploy a performance measurement system. At the lower layer technological know-how is required to support data collection and distribute data to the related persons. The second layer is where KPIs are defined, collected, monitored and analysed. The top layer, simply the objective, is to improve the business (i.e., product, process or system).

However, this is not a one-shot approach. There is a continuous cyclical process. This cycle focuses on the internal processes (diagnostic measures) and external outcomes (strategic measures). The system's control is based on KPIs that are tracked continuously over time to look for trends, best and worst practices and areas for improvement.

Figure 2. Loop in a Performance Measurement System

Define KPIs

Monitor KPI Compliance

Gather and Analyse Data

Refine the KPIS

Improve Product/Service, Process, System

Technological Know-how (Databases, Intranet, O/S, etc.)

Performance Measurement System (Define KPIs, Collect and Analyse Data)

Business Improvement (Achieve the Objective)

Prepare Prerequisites



3. ACTIVITIES

3.1 Activity: Prepare Prerequisites

Activities given in this section are optional. They are given in order to illustrate the complete picture. Ideally implementation should begin with identifying external customer segments and determining what they want. Then a strategy should be developed which will lead to defining the organisation’s key deliverables, KPIs and core production processes. KPIs should then be implemented in the core processes according to where the greatest return or risk exists. Though this sequence is conceptually sound it is not always necessary to have a well-defined strategy before implementing KPIs. Since everybody wants whatever they are buying cheaper, better and faster. There are usually significant areas of opportunity available that can be addresses with KPIs while any strategic questions are being resolved [8].

Purpose: To prepare optional prerequisites. Steps: 1. Identify key customers and their requirements via activity Identify Customer Requirements. 2. Develop vision, mission and strategy of the organisation via activity Develop Organisation

Mission, Vision and Strategy. 3. Create maps of core processes via activity Identify Core Processes. Input Artefacts: • All customer related data: Customer

comments, surveys, interviews, etc.

Resulting Artefacts: • List of customers and customer segments • Customer requirements • Vision and mission statement • Strategy statement • Maps of Core processes

Frequency: Once initially. Role: Quality Manager, Business Analyst. Tips and Tricks: None.



3.1.1 Activity: Identify Customer Requirements Purpose: To identify key customers and their requirements. Steps: 1. Create the list of products and/or services provided by your organisation. 2. Identify your major (segments of) customers and their profiles (i.e., services, products,

delivery methods) 3. Review available data on customer needs, comments, surveys, etc. 4. Draft customer requirements. 5. Validate customer requirements with stakeholders. 6. Refine and revise customer requirements. 7. Prioritise customer requirements. Input Artefacts: • All customer related data: Customer

comments, surveys, interviews, etc.

Resulting Artefacts: • List of customers • List of customer segments • Customer requirements

Frequency: Once initially. However it should be revised after a change in the system, products, processes, etc. Role: Quality Manager, Business Analyst. Tips and Tricks: • If possible, requirements should specify a level of acceptable and not acceptable

performance. • Requirements should depend on (or be validated) by the Voice of the Customer. • For step 2 you may use the following forms:

o Who are our customers? o Customer Background Information o Customer Evaluation Form o Customer Segments



3.1.2 Activity: Develop Organisation Mission, Vision and Strategy Purpose: To develop vision, mission and strategy of the organisation Steps: 1. Develop Vision and Mission Statement. It is a declaration of organisational purpose, what

basic social, political and industrial needs was the organisation formed to meet and what makes the organisation distinctive or unique.

o Vision: the desired future of the organisation, integrating the needs of both staff and users.

o Mission: justifies the existence of the institution in the eyes of the users. The mission explains the scope and goals of the institution's services to its users

2. Develop Values Statement, Strengths and Weaknesses. It generally covers the following: o Values that describe the culture, philosophy and approach to addressing the mission. o Strengths and weaknesses to determine reasonable strategic objectives.

3. Develop Strategic Alternatives. A listing of no more than four or five key strategies to pursue over the course of the plan, aimed at achieving the organisation’s mission.

4. Develop Strategic Plan. How will each strategy be accomplished, by when, using what resources and what measurements (if possible).

Input Artefacts: • List of customers • List of customer segments

Resulting Artefacts: • Mission statement • Vision statement • Strategic alternatives

Frequency: Once initially. However it should be revised after a change in the vision, mission or strategy. Role: Executive Management Tips and Tricks: • You may use Brainstorming and Affinity Diagram in all steps of this activity. The following

categories can be used to facilitate idea generation: o Customer satisfaction: What will create customer loyalty? o Core competency: How can we become a leading organisation? o Growth and Profitability: What would it take to make the most profitable

organisation? o Employee satisfaction: What will make more fun and exciting job for employees? o Current and Future Competitiveness: What could we do that will leave out

competitors eating our dust? • In the second step you may also consider using Force Field Analysis or the SWOT Analysis. • The final version of mission statement must be agreed to by consensus.



3.1.3 Activity: Identify Core Processes Purpose: To identify core processes Steps: 1. Identify list of core processes used in the production and service provision. 2. Create process maps for core processes via the Activity Identify Process. Input Artefacts: • Customer requirements

Resulting Artefacts: • Maps of core processes

Frequency: Once initially and before improvement of core processes. Role: Quality Manager, Business Analyst Tips and Tricks: • How to Distinguish Core and Support Processes [10]: For any organisation, certain

activities are essential. The following list may help you to cover some of the important core processes:

a) Customer Acquisition: The process of attracting and securing customers for the organisation.

b) Order Administration: Activities used to interpret and track requests for products and services from customers

c) Order Fulfilment: Creation, preparation and delivery of the order to the customer. d) Customer Service and Support: Activities ensuring customer satisfaction after

delivery of an order. e) New Product and Service Development: Conception, design and launch of new

value-adding services to customers There are also standard processes that provide key resources or capabilities that enable the core processes to perform. They are more specific than core processes:

a) Capital Acquisition: Provision of financial resources for the organisation to perform its work and execute its strategy.

b) Asset Maximisation: Deployment of capital to create the greatest possible return in alignment with the organisation’s strategy.

c) Budgeting: The process of deciding how funds will be allocated over a period of time. d) Recruitment and Hiring: Acquisition of people to do the work of the organisation. e) Evaluation and Compensation: Assessment and payment of people for the

work/value they provide to the company. f) Human Resource Support and Development: Preparation of people for their current

jobs and future skill/knowledge needs. g) Regulatory Compliance: Processes ensuring that the organisation is meeting all laws

and legal obligations. h) Facilities: Provision and maintenance of physical plant and equipment so that the

business can perform its functions. i) Information Systems: Movement and processing of data information to expedite

business operations and decisions. j) Functional and/or Process Management: Systems and activities to ensure effective

execution of the work of the business.



3.1.4 Activity: Identify Process Purpose: To chart the current activities involved in achieving a particular service or product and their interrelationship and identify all of the components of the current process. Steps: 1. Determine activities involved in achieving a particular service or product and write them in

“sticky notes”. 2. Establish the dependencies of each activity to others and move “sticky notes” to define logical

sequences of activities. 3. Determine activities that can be executed in parallel to other activities or sequences and

move “sticky notes” to illustrate these parallel relationships. 4. Add lines with arrows to connect activities and sequences and illustrate the flow of activity. 5. [Optional] For clarity, realign “sticky notes” to place the 'primary' sequence (the one to which

other activities and sequences contribute) in a straight line across or down the overall chart. 6. [Optional] Add to each activity or sequence the name of the party responsible for its

completion. Input Artefacts: None

Resulting Artefacts: • Map of process

Frequency: Before selecting KPIs for the process and process improvement. Role: Quality manager and responsible person(s) for this process. Tips and Tricks: • This technique is frequently used during Business Process Re-engineering to establish the

current situation. It could also be used to illustrate a proposed new process prior or during the planning and implementation stage of a change situation.

• You may use a process-mapping technique to document your activity: Block Diagram, Flowchart, SIPOC Map or System Modelling.

• In step 6 make sure that a role is used while naming responsible party, not a person (e.g., use “Business Analyst” instead of “John Doe”).



3.2 KPI Definition

3.2.1 Activity: Define KPIs Purpose: To generate a possible set of KPIs covering system, customer satisfaction, human factors, product/services and processes. Steps: 1. Create a possible list of KPIs for the following categories: financial, customer satisfaction,

human factors and business processes by Brainstorming or another appropriate idea-generating technique.

2. Formulate each KPI by using the KPI Design Form as a result of a brainstorming meeting. 3. Apply KPI Checklist on the possible KPIs (result of step 1) in order to select your initial set of

KPIs. 4. Refine data collection methods for each KPI via the activity Develop Data Collection Plan. 5. [Optional] Set a baseline and an objective for each selected KPI (if you have historical data). 6. Implement data collection procedures. 7. [Optional] Test data collection procedures. Input Artefacts: • Customer requirements • Process maps • Introduction to KPI Models • KPI Portfolio • KPI Top 12

Resulting Artefacts: • List of possible KPIs • List of selected KPIs

Frequency: Once initially and after a change in the system, products, processes, etc. Role: Quality manager. Tips and Tricks: • You may use Brainstorming and Affinity Diagram in step 1. • If appropriate, during Brainstorming break the issue down into manageable topics, e.g. 'the

deployment of KPIs ' might split into 'how to generate the right KPI specifications', 'how to measure KPIs’, 'how to report KPIs' and 'how to interpret KPIs and use for improvement’.

• Brainstorming meeting is organised by the quality manager.



3.2.2 Activity : Develop Data Collection Plan Purpose: To define a standard operating procedure for data collection. Steps: 1. [Optional] If applicable, decide which stratification factors are most important to analyse data

and/or to find the causes of problems. 2. [Optional] If applicable, develop a sampling scheme. 3. (What) Identify the type of data that will be collected or the 'event' to be tracked. 4. (How) Determine means of data collection. Examples might be:

• A plan of the layout of the area in which the 'event' is occurring; • An illustration of the product incurring the 'event' (defect); • An example of the product itself; • A process-map of the process within which the 'event' is occurring; • A checklist listing the zones within the overall organisation in which the 'event' is

occurring. 5. (When) Describe the appropriate time interval and frequency to be used in data collection. 6. (Where) Describe where the data will be collected. 7. (Who) Determine who is best placed to collect the data. Input Artefacts: • KPI for which data to be collected • KPI Measurement, Monitoring and Analysis

Guide

Resulting Artefacts: • KPI Data Collection Plan

Frequency: For each selected KPI. Role: Quality manager and/or responsible person(s) where data is collected (i.e., a process, system, etc.) Tips and Tricks: • Use the form Data Collection Plan in order to formalise your work. • Data stratification is generally done according to the following factors [10]:

a) Who (Examples: department, individual, customer type, etc.) b) What (Examples: type of complaint, defect category, reason for incoming call, etc.) c) When (Examples: quarter, month, day of week, time of day, etc.) d) Where (Examples: region, city, specific location on product: top right corner, on/off

switch, etc. • The accuracy of data collection is paramount while determining the data collection method

and care should be exercised when determining how and by whom data is to be collected particularly when large geographical areas are under examination.

• In step 4, you may use Checksheet for data-gathering and interpretation tool. • In case of a process, the interval in step 5 will be determined by the cycle time of the process. • For processes, process maps created in the activity Identify Process help the team determine

who should collect data and where data should be collected in the process.



3.3 Monitoring and Analysing Data

3.3.1 Activity: Monitor KPI Compliance Purpose: To collect data for the KPIs as stated in the KPI Data Collection Plans and monitor KPI compliance. This activity is the part of the operational control. Steps: 1. Record the event data as appropriate –stated in the KPI Data Collection Plans- until such

time as an adequate sample has been obtained. 2. [Optional] Set a baseline. The main question while determining baseline is “what

performance level can be maintained for this KPI?” In order to determine this performance level it may be necessary to select a period that seems to represent normal operating conditions and derive a performance level from this period.

3. [Optional] Set the KPI objective. This is desired/targeted performance level for the KPI. Be sure that the KPI objective is an attainable and affordable value.

4. [Optional] Check whether the KPI objective is attained. If the objective is not attained, evaluate data to determine action to resolve the root cause of the problem.

5. [Optional] Communicate results with stakeholders Input Artefacts: • KPI Data Collection Plans • KPI Measurement, Monitoring and Analysis

Guide

Resulting Artefacts: • Operational Monitoring Reports

Frequency: Depending on the criticality and stability of the operations. Usually daily or weekly. Role: Operational Manager Tips and Tricks: • In step 1 data sample size is critical. As a general rule, if in doubt - increase the sample size. • While setting baseline and objective for a KPI a realistic tolerance level must be included if

you do not sufficient historical data. • In step 4 first try to use collected data (i.e., facts) to find “the root cause”. Run Charts, Control

Charts, Histograms and Scatter Diagrams are the most popular charting techniques used to identify the problems and causes. If collected data does not help, use a creative problem-solving technique like Brainstorming, Cause-Effect Diagrams, Interrelations Digraph, etc. to determine the root cause.

• In step 4 consider monitoring the KPI (or measurements contributing to the KPIs) for stability (i.e., variance) and trends.

• KPIs do not exist in a vacuum. They are affected by anything that affects an organisation or its production processes. Weather, strikes, supply line disruptions, unusual customer requests, competitors’ actions and many others can cause large deviations in the KPIs. That is why it is a good practice to note significant changes in environmental factors or unusual circumstances on charts and reports when they occur. Besides explaining what caused particular behaviour, these notes can help managers predict what will happen under similar conditions in the future.



3.3.2 Activity: Gather and Analyse Data Purpose: Gather KPI measurements and analyse them. Steps: 1. Gather KPI measurements. 2. Create executive reports containing vital information about the performance. 3. Check overall KPI compliance. 4. Analyse each KPI and determine problematic components (a component may be one of the

following: product, service, process, system, resource or expansion of market). 5. Check whether an improvement is required and possible for problematic components. 6. Prioritise and select the component(s) to be improved and execute the activity Improve

Component for the selected component(s). 7. [Optional] Communicate results with stakeholders. Input Artefacts: • KPI Measurement, Monitoring and Analysis

Guide

Resulting Artefacts: • Executive Reports • Selected Components for Improvement

Frequency: Depending on the criticality and stability of the operations. Usually monthly, quarterly or annually. Role: Quality Manager. Tips and Tricks: • For step 6 you may use the form Improvement Selection Checklist to verify whether a correct

component is selected. • You may use Decision Matrix, Pairwise Ranking or a similar tool during prioritisation and

selection in step 6. If possible priority decisions should be supported with the following: o Potential risk, o Investment required, o Payback period, o How well projects support strategic objectives, o Availability of resources, o etc.

• In step 7 you may consider using Quality Assurance Storytelling as a means of communication.



3.4 Activity: Improve Component Purpose: Improve a selected component. Component can be one the following: product, service, process, system, resource or expansion of market. Steps: 1. Establish a well-defined improvement objective. 2. Organise a team to improve the component. This involves selecting the right people for the

team; identifying the resources available for the improvement effort, such as people, time, money and materials; setting reporting requirements and determining the team’s level of authority.

3. Identify the root causes that prevent the component from meeting the objective. The team begins the Plan-Do-Check-Act Cycle here, using the Cause-and-Effect Diagram or Brainstorming tools to generate possible causes why the component fails to meet the desired objective.

4. Develop an improvement plan for implementing a change based on the possible reasons for the component’s inability to meet the objective set for it. These root causes were identified in step 3.

5. [Optional] Check whether KPIs related to the component do exist and provide all the required data for the evaluation of the improvement objective. If not, define new KPI(s) for this component and collect baseline data.

6. Test the changed component and collect data. 7. Assess whether the changed component is stable. The team uses a Control Chart or Run

Chart to determine stability. If the component is stable, the team can move on to step 8; if not, the team must return the component to its former state and plan another change.

8. Assess whether the change improved the component. Using the data collected in step 6 and a Histogram, the team determines whether the component is closer to meeting the process improvement objective established in step 1. If the objective is met, the team can progress to step 9; if not, the team must decide whether to keep or discard the change.

9. Determine whether additional improvements are feasible. If yes, re-start the activity from step 3.

10. Standardise the change and reduce the frequency of data collection Input Artefacts: • Selected Component for Improvement • List of KPIs related to the component

Resulting Artefacts: • Improved component

Frequency: For each improvement Role: Quality Manager Tips and Tricks: • You may use the form Improvement Plan in order to plan and document your improvement

effort.



Figure 3. Flowchart of Activity “Improve Component”

No

Yes

Yes

No

Yes

No

Yes

Yes

Step 9More Improvement?

Step 1Establish the improvement

objective

Step 2Organise the right team

Step 6Test the change and collect data

Step 3Identify root causes for lack of capability

Step 4Plan to implement the change

Step 5Modify the KPIs (if necessary)

Step 8aImproved?

Step 7Stable?

Step 10Standardise the change and reduce

the frequency of data collection

No

Step 8b Keep?



3.5 Activity: Refine KPIs Purpose: To refine KPIs (i.e., addition, modification, deletion). Steps: 1. [Optional] Add missing KPIs for critical components. Apply KPI Checklist for the new KPIs.

Determine data collection methods for each KPI via the activity Develop Data Collection Plan 2. [Optional] If necessary, refine data collection methods for each KPI via the activity Develop

Data Collection Plan after a change in the associated component. 3. [Optional] Delete KPIs that are not providing required feedback. 4. [Optional] Set/Modify KPI baselines and objectives. Input Artefacts: • List of current KPIs • KPI Data Collection Plans

Resulting Artefacts: • Modified KPI list • Modified KPI Data Collection Plans

Frequency: Not applicable Role: Quality Manager Tips and Tricks: None



4. CONCLUSIONS AND FUTURE WORK

This document presents general framework for a performance measurement system. The ultimate goal of the performance measurement is to ensure customer satisfaction by providing high quality services/products in a cost effective manner. The deployment of a performance system can be quite costly depending on the number of Key Performance Indicators (KPIs) and the effort required to measure them. Therefore, selecting the appropriate KPIs is the key for successful application of KPI deployment and common sense must prevail when selecting KPIs.

The guide and forms associated with it are proposals. Therefore, it is likely that this document will evolve with your feedback.



- Page left intentionally blank -



APPENDIX A KPI DESIGN FORM

You may use the following form to design your KPIs:

KPI DESIGN FORM KPI Name Description

Reason

Type Quantitative Qualitative Effort Low Medium High Unit Assessment Method

Tools

Day Week Month Analysis Frequency Quarter Year Year+ Comments



APPENDIX B MAJOR PITFALLS

Major Pitfalls of Performance Measurement Systems are given in [6]:

Measures not linked to strategy - Critical to do initially, but also revisit when either the organisational strategy or structure changes

Measures not driven into organisation(s) below: Breaks the link with overall strategy. Should be driven into staff performance agreements at all appropriate levels.

Too many measures: Creates lack of focus on what is really critical to managing your business.

Not enough critical measures: You could be missing information vital to operations.

Focusing only on the short-term: A cross-section of past (lagging), present and future measures is critical.

Conflicting measures: Sub-optimises staff or organisational performance. Example: Measuring reduction of office space per staff member while also measuring staff satisfaction with facilities.

Measuring progress too often: Could result in unnecessary effort and excessive costs, resulting in little or no added value

Not measuring progress often enough: May not know about potential problems until it is too late to resolve easily.

Collecting too much data: Could result in a mountain of data that really doesn’t tell us anything more than a lesser amount of the same data.

Collecting inconsistent, unrepresentative or unnecessary data: Critical to understand what the data will look like, when it will be collected, at what frequency, by whom and what it means, up front.

Reducing the value of data: Too much data roll-up (summary) can mask the impact of potentially significant events or trends

Driving the wrong performance: Be careful that the measure(s) you select will end in the desired result.

Encouraging competition and discouraging teamwork: Measuring vertically frequently pits one internal organisation against the others. Try to measure horizontally.

Failure to base business decisions on data: Developing performance measures or collecting data only to comply with a requirement does nothing to improve the position of the company.



APPENDIX C KPI CHECKLIST

Here is the KPI checklist you may use to select each KPI:

KPI CHECKLIST KPI Date

Place a checkmark next to all of the items that apply to your KPI.

Does the KPI motivate the right behaviour?

Is the KPI measurable?

Is the measurement of this KPI affordable (cost-effective)?

Is the KPI objective (i.e., targeted performance level) attainable?

Are the factors affecting this KPI controllable by you?

Is the KPI meaningful?

Note: If you have selected an appropriate KPI, you should be able to check all of the items above.

The selection process is complicated by the number of potential KPIs and must be tempered by considerations such as:

• Organisational experience with KPIs

• Type of behaviours to be motivated

• Cost and effort of collection.

Common sense must prevail when selecting KPIs. It should be remembered that the goal is to ensure customer satisfaction by providing high quality services/products in a cost effective manner.

To meet above goals, AIS organisations should consider the following principles while selecting or defining new indicators:

• A KPI must motivate the right behaviour

• A KPI must be measurable

• A KPI must be affordable

• The objective set for an KPI must be attainable

• Factors effecting the indicator must be controllable by the service provider

• A KPI must be meaningful to all the parties.



Indicators that motivate the right behaviour: When choosing KPIs, it must be first focussed on the behaviour that is wanted to be motivated. There are no hard and fast rules for this; first a KPI must reflect the performance at the right areas:

• Processes and factors most critical to business

• Processes and factors where most improvement needed

• The most dissatisfied client group

• The most politically powerful client group

Then it must motivate a pre-defined behaviour in these areas:

• To reduce costs and/or defects

• To increase production

• To speed up time-to-market

• Etc.

An initial set of KPIs must be picked up in order to measure performance related to these behaviours.

Measurable: If it is not possible to measure a KPI, that KPI is worthless and should not be considered. As an example user (i.e., end-to-end) response time is a very popular KPI among users. Certainly, this is one of the key factors shaping the users’ opinions about the quality of service they receive. Unfortunately, measuring user response time on an end-to-end basis is still a technical challenge today. At the other end of the feasibility spectrum is the availability of a service. This is relatively straightforward and can be measured with a minimum effort and difficulty.

Affordable: No organisation has unlimited resources. The amount that can be spent on delivering any service or product is limited. Therefore, in setting objectives for a KPI, it is also necessary to consider whether the desired objective (i.e., targeted performance level) is affordable. This might also be thought of as being cost effective. A KPI should not require a heavy investment of time and money. In some cases, it will be necessary to devise alternative KPIs if the required data is not easily obtainable.

Another important issue in this topic is the easiness of data collection. If KPIs cannot be easily measured, then they will quickly lose favour, and eventually be ignored completely. No one is going to spend an excessive amount of time to collect KPIs manually. Ideally, all KPIs will be captured automatically, in the background, with minimal overhead.

Attainable: To be useful, the KPIs must be set to reasonable, attainable objectives. It may be difficult to select an initial, appropriate setting for a KPI, especially when there is no available historical record of those KPIs. Organisations with active KPI programs will have the data needed to set a proper baseline. Others will have to perform an initial assessment to establish that baseline. Unless strong historical measurement data is available, it should be expected to re-visit and re-adjust the



setting at a future date. In addition, a realistic tolerance level must be included while setting the objectives if there is no historical measurement data.

Controllable: A KPI must represent something that is controllable. That is, the service provider must have the ability to exercise control over factors that determine the level of service delivered. The service provider has control over bringing an AIS publication to the post office, but has no control over the speed by which the post office delivers the mail. Thus, an objective that "all AIS publications will be delivered to our clients within 48 hours after production completion" is unfair and likely to be de-motivating to the service provider.

Examples for these issues can be extended very easily:

• In a country where there are frequent power failures and the budget of the AIS organisation does not permit the purchase of a standby generator to prevent service interruptions during these failures,

• Strikes by workers,

• Poor service by a telecommunication company without a reasonable alternative.

These are just a few of the factors that can place certain objectives behind the control of the service provider.

Meaningful: In the context of the service level agreement a KPI must be meaningful and understandable to all the parties (i.e., client and service provider) - another way of stating this is to say that it must be relevant.

Lets take an AIS organisation, which provides some of its services/products via Internet, as an example. Service provider decides to use packet collisions and dropped packets in order to illustrate the availability of its LAN component. They can be easily captured however they mean little to anyone other than a network engineer or system administrator. Although these KPIs might impact on the availability from the client’s perspective, the relevance of such KPIs to the service they receive is quite difficult to grasp.



APPENDIX D CUSTOMER FORMS

D.1 Who Are Our Customers?

The ultimate goal of any improvement effort is to better satisfy the needs of the customer. Thus, before formulating any improvement goals, the organisation must first assess who their customers are. Thinking about customers can raise fundamental questions about the purpose and mission of the organisation.

The following questions from Deming are useful to stimulate thinking about customers:

Who makes the decisions about whether to buy your product or service?

How do you distinguish between quality as your customer perceives it and quality as your managers and work force perceive it?

How does the quality of your product, as your customer sees it, agree with the quality that you intended to give him/her?

Do your customers think that your product lives up to their expectations?

What do you know about the problems of your customers in the use of your products? What tests do you make of your products in service?

Do you depend on complaints from customers to learn what is wrong with your product or service?

Are your customers satisfied with the service that you provide? If yes, what is satisfactory about it? How do you know?

Will your customers of today be your customers a year hence? Two years hence?



WHO ARE OUR CUSTOMERS? Customer Products/Services Used



D.2 Customer Segments

Since most organisations have many customers, it is typical for organisations to separate customers into smaller subgroups, often referred to as segments, using a variety of methods. Most organisations segment customers based on what products and/or services they use. Customer segmentation assists in defining the requirements of different groups. It also provides a basis for understanding similarities among customers. It also clarifies which customers may be impacted by improvement efforts focused on particular products or services.

CUSTOMER SEGMENTS SEGMENT

CUSTOMERS

SEGMENT

CUSTOMERS

SEGMENT

CUSTOMERS

SEGMENT

CUSTOMERS

SEGMENT

CUSTOMERS

SEGMENT

CUSTOMERS



D.3 Customer Evaluation Form

This form should ideally be filled after customer interviews. However, if interviewing customers is too costly simply fill the form with your perception.

CUSTOMER EVALUATION FORM Date Customer Contact Person Tel Fax E-mail Postal Address

What products and services are currently provided to this customer?

What are the most important features or characteristics of the products/services provided to this customer?

What are the major delivery methods used for this customer?

What aspects of products/services is the customer satisfied with?

What needs improving?

Additional Comments and Observations



D.4 Customer Background Information

This form is to be used before customer interviews and/or evaluation.

CUSTOMER BACKGROUND INFORMATION Date Customer Contact Person What products and/or services has this customer acquired or used from your organisation in the past?

How often does this customer acquire products/services from us?

How long has this customer been using our products/services?

How much of our budget is related to products/services for this customer?

Does this customer have any pattern in the acquisition or use of our products/services?

Does any complaint data exist to help clarify customer requirements?

Does other customer satisfaction data exist?

Does this customer refer other organizations to you? Who?



APPENDIX E IMPROVEMENT FORMS

E.1 Improvement Plan

IMPROVEMENT PLAN A. Statement

Selected Component Date Description

System Process Product Type of Component

Service Resource Expansion of Market Improvement Goal(s)

Area of Concern

Customer Impact

Products and Services Effected

Stakeholders



B. PLANNING Resources Human

Training

Financial

Other

Reporting Requirements

Suggested Timeline

C. MEASUREMENT REQUIREMENTS Existing KPIs Name Description

KPIs to Develop Name Description



E.2 Improvement Selection Checklist

This form may be used for each component that is considered for improvement.

IMPROVEMENT SELECTION CHECKLIST Component Date

State problems or expectations identified for this component

1

2

3

4

Place a checkmark next to all of the items that apply to your component.

The component and its associated improvement effort can be defined. (Be careful not to pick something too big. It should be possible to complete the improvement effort within 90 days.)

A problem in the component occurs frequently. (A Pareto analysis may be helpful.) The problem area is well known and it is visible by the customer (external) or in the office (internal). Improvement of this component is important to the customer or the business. People will appreciate it if the component is improved. There is a good chance of success in improving the component. No one else is currently working on this component. Required changes can be put into effect with little or no outside help This is truly an improvement effort, not just an attempt to impose a solution on a problem

Note: If you have selected an appropriate component, you should be able to check all of the items above.



APPENDIX F DATA COLLECTION PLAN

The Data Collection Plan summarises the important information about this critical task on one form. Complete the form for each KPI for which data will be collected. It is likely that this form will be used many times.

DATA COLLECTION PLAN KPI Name: Date: What kind of data is to be collected?

How will the data be collected?

When will the data be collected?

Where will the data be collected?

Who will collect the data?

kpi deployment guide · kpi key performance indicator. mission statement a brief statement of the...

Documents