understanding metrics – what to measure, and why - john custy

#atlassian

JOHN CUSTY • MANAGING CONSULTANT • JPC GROUP • @ITSMNINJA

Understanding Metrics:What to Measure and Why

Service Management Practitioner, Consultant And EducatorAbout John Custy

• Distinguished Professional in IT Service Management

• ITIL Expert & ITIL Service Manager

• ITIL Intermediate – SS, SD, ST, SO, CSI, OSA, SOA, PPO, RCV

• KT Certified Instructor• ITIL Accredited Trainer• KCS Verified Consultant• ISO/IEC 20000 Consultant• ISFS, ISMAS based on

ISO/IEC 27002• HDI Faculty & Certified

Instructor

A story about metrics…

Source: Joëlle Flumet’s Broken Bed Installation

Source: betweennapsontheporch.net

The customer rating is 5/5, but the customer

WON’T return!

What are you measuring? Why are you measuring it?

T Y P E S O F M E T R I C S

P U R P O S E O F M E T R I C S

What to Measure and Why

C O M M O N S U P P O RT M E T R I C S

M E T H O D S O F R E P O RT I N G

Q U E S T I O N S


How to use metrics


How to use metrics• Inform your stakeholders• Report measurements so that stakeholders can understand activities and results• Promote the value of the organization• Determine the best way to communicate the information to the stakeholders• Perform better stakeholder analysis to facilitate stakeholder buy-in• Improve performance - people do what is measured


What are we trying to accomplish?

• Account for IT Processes and Deliverables

• Inform stakeholders

• Understand IT Performance

E N S U R E A L I G N M E N T

• Achieve certifications; ISO/IEC 20000, COBIT

• Measure progress to goals/objectives

C O M P L I A N C E

• Measure IT Performance

• Control IT Processes

• Maximize IT Productivity (people)

• Report Costs

• Demonstrate value of IT Organization

O P E R AT I O N A L E X C E L L E N C E


What are we trying to accomplish?

The Relationship of Metrics to Goals

The Relationship of Metrics to Goals

V I S I O N

M I S S I O N

G O A L S

O B J E C T I V E S

C S F

K P I

M E T R I C S

M E A S U R E M E N T


Sharing Accomplishments

What should you report?


Sharing Accomplishments

Key performance indicatorsCritical success factorsVariances to baseline

Progress towards targetsAnnotate milestones and abnormalities

Service improvement projects

What should you report?

Understanding the Different Types of Metrics

D I F F E R E N T T Y P E S O F M E T R I C S

Metrics & Characteristics

Quantitative

• How much or how many

• Ex. The number of times customers contact the service desk

% :)


Metrics & Characteristics

Quantitative



% :)

M E T R I C S !

!

•Performance indicators (PI) !

•Key performance indicators (KPI)!

•Key results indicators (KRI)

C H A R A C T E R I S T I C S !!

•Efficiency vs. effectiveness!

•Leading vs. lagging


Efficiency vs. Effectiveness

Quantitative



% :)


Efficiency vs. Effectiveness

Quantitative



% :)

E F F I C I E N C Y !

!

•How fast?!

•How many?!

•Transactional Cost!

•Incident/Request/Access Management!

•Departmental Goals

E F F E C T I V E N E S S !!

•Accuracy!

•Customer Satisfaction!

•Total Organizational Cost!

•Problem Management!

•Enterprise Objectives


Quantitative vs. Qualitative

Quantitative



% :)


Quantitative vs. Qualitative

Quantitative



% :)

Q U A N T I TAT I V E !

•How much or how many!

•Ex. The number of times customers contact the service desk

Q U A L I TAT I V E !

•How well something or someone is performing.!

•Ex. Customer Satisfaction, Employee Satisfaction, stock price.

Service Metrics

• Time to process an order • Time to check inventory item • Time to send/receive an e-mail • Time to …

@ITSMNinja

End-to-End Performance

• Cost Per Transaction • Cost Per User

@ITSMNinja

End-to-End Performance

Uptime – compared to …• Downtime per Service

• Frequency and total amount of time• Number of incidents (type/category)• Number of recurring incidents• Time per incident

@ITSMNinja

Service Availability

• Problems identified per Service • # incidents per problem• Lost time per problem

• Changes Per Service • # successful changes (time, budget)• Lost time due to changes – incidents and requests• # Service Requests due to changes• # problems due to changes – IT and business lost time• % improvements due to changes

@ITSMNinja

Service Availability

B A L A N C E N E E D E D

Operational metrics allow you to understand where to improve

Service metrics report on the overall performance

of the service

What Type of Metrics Are Reported?

What Type of Metrics Are Reported?

I T I N T E R N A L M E T R I C S

S E N I O R M A N A G E M E N T

S E RV I C E M A N A G E M E N T

B U S I N E S S U N I T M E T R I C S

R E G U L AT O RY /C O M P L I A N C E

Four Types of Process Metrics

P R O G R E S S E F F I C I E N C Y E F F E C T I V E N E S S C O M P L I A N C E

!I N P R O C E S S

M AT U R I T Y

!U S E O F

R E S O U R C E S

!C O R R E C T A N D

C O M P L E T E T H E F I R S T T I M E

!T O P R O C E S S A N D

R E G U L AT O RY R E Q U I R E M E N T S

Identifying Common Service

Desk Metrics

• Response • % connected immediately (Real-Time)• Abandon Rate• Wait (hold/queue) Time• Average Speed to Answer (ASA)• Response Time service level XX% in YY seconds• Call-Back Time• Desktop (PC)

@ITSMNinjaTypical Operational Measurements

• Resolution • Resolved First Contact• Resolved X hours, Y hours, Z hours• Cases re-opened, Repeats• Requests resolved without assistance (self-help)• Calls/Cases avoided due to self-help


• Cost • Call/Contact• Customer/User• Total Cost (TCO) of Support/Service


• Service Desk • Volumes, trends• Performance to goals

• Incident Management • Volumes, trends, repeat incidents• Reduction in restoration time• Performance to goals• Customer Satisfaction


• Request Fulfillment • Volumes, trends• Time to complete requests• Performance to goals• Customer satisfaction


• Customer Satisfaction • Employee Satisfaction


@ITSMNinjaTypical Operational Measurements• Knowledge Base Usage • Accesses/Searches per contact• # solutions per search• # solutions searched/opened/viewed• Time spent reviewing solutions• Ease of finding solutions• Quality of solutions (ability to use solutions)

• Service Asset and Configuration Management • Errors in CMDB• Resources improvement utilizing CMDB

• Change Management • Number of incidents/requests due to the change• Additional (reduction) workload due to changes

• Release Management • Number of incidents/requests due to the release• Additional (reduction) in workload due to releases


S E RV I C E D E S K M E T R I C S


W O R K L O A D

•

• Vo l u m e s • C a l l s / C a s e s p e r

c u s t o m e r p e r m o n t h

• N u m b e r o f re g i s t e re d u s e r s /To t a l n u m b e r o f u s e r s

• T i m e s p e n t c o n t a c t i n g u s e r s

• T i m e s p e n t o n c h a n g e re l a t e d i n c i d e n t s / re q u e s t s


W O R K L O A D

•






I N D I V I D U A L S

• N u m b e r o f c a l l s t a k e n

• Av e r a g e H a n d l e T i m e ( A H T )

• Av a i l a b i l i t y • O c c u p a n c y • N u m b e r o f

i n c i d e n t s / re q u e s t s c l o s e d o n f i r s t c o n t a c t

• C u s t o m e r S a t i s f a c t i o n

• C o n t r i b u t i o n t o K n o w l e d g e b a s e


W O R K L O A D

•













C U S T O M E R S


• F re q u e n c y o f s u r v e y i n g , N u m b e r n o t re s p o n d i n g

• Vo l u m e s • C a l l s / C a s e


W O R K L O A D

•













C U S T O M E R S


• F re q u e n c y o f s u r v e y i n g , N u m b e r n o t re s p o n d i n g

• Vo l u m e s • C a l l s / C a s e

R E S P O N S E

• Av e r a g e S p e e d t o A n s w e r ( A S A )

• % c a l l s a n s w e re d l i v e v s . . q u e u e d

• C a l l b a c k t i m e • A b a n d o n R a t e

( A B A ) • R e s p o n s e s w i t h i n

S e r v i c e L e v e l & O u t s i d e s e r v i c e l e v e l

I N C I D E N T, R E Q U E S T A N D A C C E S S M A N A G E M E N T


R E S O L U T I O N

•

• I n c i d e n t c l o s u re ( f ro m t i m e o f s u b m i s s i o n )

• M e a n T i m e f o r S e r v i c e R e s t o r a t i o n ( M T S R ) f o r L e v e l s 1 , 2 , & 3

• I n c i d e n t s m a t c h e d ( K E )

• I n c i d e n t s R e -O p e n e d

• C l o s e d F i r s t C o n t a c t

• E s c a l a t i o n s f o r re s o l u t i o n

• R e m o t e t o o l u t i l i z a t i o n

• D e s k - s i d e v i s i t s • I n c i d e n t s c l o s e d

v i a s e l f - h e l p


R E S O L U T I O N

•










V O L U M E

• To t a l n u m b e r o f i n c i d e n t s / re q u e s t s ( b y p r i o r i t y & c a t e g o r y )

!• S e c u r i t y re l a t e d

i n c i d e n t s


R E S O L U T I O N

•










V O L U M E



i n c i d e n t s

R E S P O N S E T I M E

• S e r v i c e D e s k p e r f o r m a n c e

!• L e v e l 2 / 3 – s a m e

a s S D m e t r i c s


R E S O L U T I O N

•










V O L U M E



i n c i d e n t s

R E S P O N S E T I M E

• S e r v i c e D e s k p e r f o r m a n c e

!• L e v e l 2 / 3 – s a m e

a s S D m e t r i c s

E S C A L AT I O N

• T i m e t o e s c a l a t e !

• % E s c a l a t e d t o c o r re c t g ro u p !

• Te c h n i c a l & H i e r a rc h i c a l


C U S T O M E R S AT I S FA C T I O N

• I n c i d e n t , R e q u e s t & A c c e s s M a n a g e m e n t p ro c e s s e s

•

• N u m b e r o f u n i q u e u s e r s !

• Av e r a g e t i m e p e r u s e r !

• # p a g e s v i e w e d

S E L F - S E RV I C E

Methods to Report Performance

P E R F O R M A N C E R E P O RT S

Methods to Report Performance

B A L A N C E D S C O R E C A R D

S U P P O RT S C O R E C A R D

Factors to Consider When Reporting

Factors to Consider When Reporting

• Who are the stakeholders? • How does what you are reporting impact the stakeholders? • Reports must be easy to read and understood, thus they need to be developed with

the stakeholder in mind.• Reports need to show how the support center is contributing to the goals of each

stakeholder and the business.• Reports must identify the appropriate channels to communicate with each of the

stakeholders.

Daily Report Examples

Trending Report Examples

Activity Report Examples

Present your achievements with

scorecards

Scorecards Drive Transformation

High (Business Transformation)Low (Process Improvement) Moderate

I m p r o v e m e n t G o a l

Rel

evan

t to

IT

Ho

li

st

ic

Gen

eral

Spe

cific

T C OISO/IEC17799

ITIL/ISO/IEC20000 C M M

COBIT

Six Sigma

ISO/IEC 9000

Malcolm Baldrige Award

Scorecards

Standards help us to ensure that IT is aligned to meet business objectives

• Simple indicator • Reference base • Measures the key issues • Reports on progress to goals

@ITSMNinjaSupport Scorecard

Scorecard Criteria: Operational Performance

Overall performance

Response time

Resolution time

Closed first call

Abandon time

Wait time

Status time

Backlog aging

C O L U M N T I T L E C O L U M N T I T L E C O L U M N T I T L E

Scorecard Criteria: Operational PerformanceSERVICE SCORECARD The Acme Support Center Scorecard provides a weekly report of performance on our Service Level commitments.

CS Overall Support Center Product

Key Service Area Goal Actual

Response Time Front-line Measurements:• Call Pick-Up Time - All incoming calls are answered by a support consultant• Call Waiting Time - Average is less that 3 minutes• Back-line Measurements:• Non-Accepted Call Back Time - All customers not responded to on the initial

call by a support consultant will be called back within 30 minutes.

80%3 minutes

90%

Resolve Time Resolved on First Contact - 30% resolved first call (4 month goal is 50%)Resolved Same Day - 40% resolved within 1 business day (4 month goal is 60%)Resolved Same Week - 85% resolved within 5 business days

30%40%85%

Status Priority 1 Issues - Customer provided status every 4 hours until resolvedPriority 2 Issues - Customer provided status every 24 hours until resolved or workaround providedCall Aging - Manage backlog so that no more than 20% over 2 weeks and 5% over 30 days

80%80%80%

Backlog (Average age of open items) 3 days10 days

Event Survey Overall satisfaction rating on a 1-5 scale 4.1

C O L U M N T I T L E

Service Desk Scorecard

Overall performance

Response time

Resolution time

Closed first call

Abandon time

Wait time

Status time

Backlog aging


Customer sat/quality

Overall satisfaction

Response satisfaction

Resolution satisfaction

Status satisfaction

Improvement goals

Alignment to business


Acme Service Desk Scorecard

Balanced Scorecard

Business Goals

Financial PerspectiveProvide a good return on investment on IT-enabled business investments

Manage IT-related business risks

Improve corporate governance and transparency

Customer Perspective

Improve customer orientation and service

Offer competitive products and services

Establish service continuity and availability

Create agility in responding to changing business requirements

Achieve the cost optimization of service delivery

Obtain reliable and useful information for strategic decision making

Internal Perspective

Improve and maintain business process functionality

Lower process costs

Provide compliance with external laws, regulations and contracts

Provide compliance with internal policies

Manage business change

Improve and maintain operational and staff productivity

Learning and Growth PerspectiveManage product and business innovation

Acquire and maintain skilled and motivated personnel


IT Balances Scorecard @ITSMNinja

Source: ITIL Continuous Service Improvement

You cannot manage what you cannot CONTROL

You cannot control what you cannot MEASURE

You cannot measure what you cannot DEFINE

What do you need to measure? What should you be doing you do with the metrics you produce?

Thank you!

John Custy • Managing Consultant • JPC Group • @ITSMNinja

Join me tomorrow @2PM to learn

Knowledge Centered Support (KCS) – The Methodology that Really Works

understanding metrics – what to measure, and why - john custy

Software