Challenges in Business Performance Challenges in Business Performance Measurement: The Case of a Measurement: The Case of a

Corporate IT FunctionCorporate IT Function

Stephen Corea & Andy Watters (Warwick University, UK)

Presentation OutlinePresentation Outline

Research Motivation

Theoretical Review

Methodology

Case Study: GITS

Findings & Discussion

Conclusion

Business performance measurement (PM) – presenting relevant information to management staff for assessing the organization's progress towards achieving strategic/operational aims

Several major PM frameworks proposed recently: field dominated by prescriptive, top-down perspective (formal derivation from strategy)

Research BackgroundResearch Background

RESEARCH AIM:

exploratory study to understand the challenges in mounting dashboard based PM practices in a corporate IS Dept

Significant no. of PM initiatives fail to take root or adequately deliver expected benefits (70% according to McCunn, 1998)

Need for greater understanding of on-the-ground challenges and bottom-up perspective of challenges involved

1. Robust PM system should take a ‘balanced’ approach (Kaplan & Norton, 1992)

2. Identification/population of useful performance measures (or metrics) to capture progress towards goal attainment is a key but not easily satisfied criteria (Neely et al, 1997):

Theory: PM ImplementationTheory: PM Implementation

Key proposed principles of PM design

Should be grounded in strategy: performance metrics must be derive from predefined strategy/objectives (De Tonia & Tonchia, 2001)

Should incorporate a high proportion of ‘leading’ indicators (Eckerson, 2006) i.e. in comparison to ‘lag indicators’

Theory: PM DesignTheory: PM Design 22 most-cited recommendations for designing

measures (Neely e al, 1997)

1. Be derived from strategy2. Be simple to understand3. Provide timely and accurate feedback4. Be actionable: based on quantities that

can be influenced or controlled5. Reflect the business process: both

customer and supplier should be involved in defining

6. Relate to specific targets7. Be relevant8. Be predictive: part of a closed

management loop9. Be clearly defined10. Have visual impact11. Should focus on improvement12. Be consistent (i.e. should maintain

significance as time goes by)

1. Provide fast feedback2. Have an explicit purpose3. Be based on explicitly defined formula

and source of data4. Employ ratios rather than absolute

numbers5. Use data which are automatically

collected as part of a process6. Be reported in a simple consistent

format7. Be based on trends rather than

snapshots8. Provide information9. Be precise – be exact about what is

being measured10. Be objective – not based on opinion

3. Systemic aspect of effective PM: process rationalisation, shared understanding & staff commitment, IT support (data capture/collection, processing & presentation)

4. Dashboards: visual impact, data quality and timeliness (Few, 2005; Dixon et al., 1999)

5. PM for the IT Function: need for spread of measures across 3 categories (Stanwick & Stanwick, 2005) – (i) efficiency; (ii) effectiveness; (iii) productivity

Theory: PM ImplementationTheory: PM Implementation

Interpretive case study method (Walsham, 1995): indepth single-site case study, aimed at theoretical generalisation

Case organisation/unit: GITS (Group IT Services), the corporate IT function of Multicorp (a pseudonym), a multi-national manufacturer of tobacco-based products

Data Gathering & Analysis

multiple site visits: June to August 2006

27 semi-structured interviews, dashboards/documents review, informal conversations

inductive analysis (Glaser & Strauss, 1967): identifying patterned regularities (common themes, issues or dilemmas)

Research MethodResearch Method

Case Study: Multicorp /GITSCase Study: Multicorp /GITS

GITS (Groups IT Services) corporate IT function, formed in 2004

transform supply of IT support across Multicorp’s worldwide business units from traditional geographically-localised model into a centralised, ‘shared services’ model

to achieve cost-savings of £100 million by 2009

Major manufacturer of cigarettes & tobacco-based products 300+ brands sold in 180 ‘end-markets’ (i.e. country-specific regions);

factories in 54 countries

stable industry competitive conditions: strategy is strongly focused on efficiencies (cost-savings) across business units (esp. support)

Case Study: GITSCase Study: GITS

World Class People

Service

Quality

Irresistible Value

Cost Savings

SDS 26

Volume

£100mCSS 4.5

100%

Supply Side

Strategy & structure

vision: irresistible value

client-facing units & management team

critical need to monitor performance

3 dashboards in use

“Two years ago GITS had less than 100 staff, the management team could sit in a room and discuss in detail operational issues throughout the department. Now we’re more than 500 strong, and are doing far more things. We haven’t a clue what is going on out there, and don’t know what operational things we should be looking at.” (Manager)

GITS: Leadership DashboardGITS: Leadership Dashboard

Performance-to-date

Customer Satisfaction

Managed Volume

Cost Reduction

e2e SLA Performance

Mean Time To Fix

Application Services Operations

Technical Services Operations

GITS: Application Services DashboardGITS: Application Services Dashboard

On Targe t

Within 10% o f Targe t 2004

More than 10% Below Target Actual Actual Budget Actual BudgetFinancial Responsible Unit

Managed Volume- Ongoing Tim £'Ms 18.2 14 0.9 2.4 10.1 14.0Cost Reduction Tim £'Ms 7.2 7 2.3 4.1 4.4 7.6Cost Avoidance Tim £'Ms 3.5 4 0.3 1.7 4.0Overhead Rate Tim % 10 8 8.0% 9.5% 8.0% 8.0

Service MetricsE2E SLA Performance David % N/A 99.5% 100.0% 98.5% 99.7% 98.5% 99.5%Timeliness of First Response David Hours N/A 1.90 0.6 1.9 2.3 1.9 1.9Problem Resolution David Days N/A 1.00 0.1 1.0 0.9 1.0 1.0

Customer PerceptionProject Evaluation David /5 N/A 3.50 4.0 3.5 4.0 3.5 3.5Customer Satisfaction Survey Simon /5 3.5 3.50 3.3 3.5 3.4 3.5 3.3Improvement Evaluation David /5 N/A 3.50 3.5 3.5 3.5

Operational EfficiencyCoverage Level Within Agreed Response TimeDan % N/A 80.0% 95% 80% 97% 80% 80%Incident Evaluation David # N/A 3.50 4.0 3.5 3.7 3.5 3.5Service Office Response Time David % N/A 95.0% 100.0% 95.0% 90.8% 95.0% 95.0%

EAS/ES Demand Fulfilment Thorsten % N/A 90.0% 98% 95% 95% 90% 90%Resource Utilisation Thorsten % 100 100.0% 113% 100% 93% 100% 100%Permanent Headcount Mike # 184 220 208 208 208 208 220

Staff Development Rating Mike /32 20 22 22 22 22 22 22Prince 2 Accreditation Mike % 62.0% 80.0% 99% 75% 97% 75% 80%Leading Through Change Participation Simon % N/A 95.0% 98% 100% 98% 100% 98%

Current Period 2005 Year To Date Current Year End Forecast

P&O

2005 Year End Budget

Driving for R

esults B

uilding

Global

On Targe t

Within 10% o f Targe t 2004

More than 10% Below Target Actual Actual Budget Actual BudgetFinancial Responsible Unit

Managed Volume- Ongoing Tim £'Ms 18.2 14 0.9 2.4 10.1 14.0Cost Reduction Tim £'Ms 7.2 7 2.3 4.1 4.4 7.6Cost Avoidance Tim £'Ms 3.5 4 0.3 1.7 4.0Overhead Rate Tim % 10 8 8.0% 9.5% 8.0% 8.0

Service MetricsE2E SLA Performance David % N/A 99.5% 100.0% 98.5% 99.7% 98.5% 99.5%Timeliness of First Response David Hours N/A 1.90 0.6 1.9 2.3 1.9 1.9Problem Resolution David Days N/A 1.00 0.1 1.0 0.9 1.0 1.0

Customer PerceptionProject Evaluation David /5 N/A 3.50 4.0 3.5 4.0 3.5 3.5Customer Satisfaction Survey Simon /5 3.5 3.50 3.3 3.5 3.4 3.5 3.3Improvement Evaluation David /5 N/A 3.50 3.5 3.5 3.5

Operational EfficiencyCoverage Level Within Agreed Response TimeDan % N/A 80.0% 95% 80% 97% 80% 80%Incident Evaluation David # N/A 3.50 4.0 3.5 3.7 3.5 3.5Service Office Response Time David % N/A 95.0% 100.0% 95.0% 90.8% 95.0% 95.0%

EAS/ES Demand Fulfilment Thorsten % N/A 90.0% 98% 95% 95% 90% 90%Resource Utilisation Thorsten % 100 100.0% 113% 100% 93% 100% 100%Permanent Headcount Mike # 184 220 208 208 208 208 220

Staff Development Rating Mike /32 20 22 22 22 22 22 22Prince 2 Accreditation Mike % 62.0% 80.0% 99% 75% 97% 75% 80%Leading Through Change Participation Simon % N/A 95.0% 98% 100% 98% 100% 98%

Current Period 2005 Year To Date Current Year End Forecast

P&O

2005 Year End Budget

Driving for R

esults B

uilding

Global

GITS: Technical Services DashboardGITS: Technical Services Dashboard

Technical Services Dashboard

Reporting Period: April VISION

ANALYTICS SUSTAINABLE ORGANISATION

On Target % Achieved Team Building SDS Your Voice 6 Sigma

Objectives 0.00% People ? ?

Signed Off On Target KCC Progress Audit Memos

Memos missing Deadlines

Strategy Risk & Control 78% 16 0

To Date £ Target Stretch Key Roles No. Filled Contractor %

Cost Reduction Resourcing 23 21 13%Cost Avoidance

Transformation Plans 0

Governance in place

TS Update Stakeholders mapped

Governance & Comms

OPERATIONAL

No. Lines Performance P&L YtoD Availability CSS

Service Lines ? AVAILABILITYE2E FIX SCOREGITS OVERALL CSS

Budget Forecast To date Variance Forecast To date Variance

Financial Ops. £2,222,333.44 £1,111,111.22 £1,233,222

No. No. Quality Improvement

Current Budgets

ForecastedExpenditure

% Projects Complete

£ Benefits

Projects

Revenue

Deliver irresistible value through...

Expenses

Technical Services Dashboard

Reporting Period: April VISION

ANALYTICS SUSTAINABLE ORGANISATION

On Target % Achieved Team Building SDS Your Voice 6 Sigma

Objectives 0.00% People ? ?

Signed Off On Target KCC Progress Audit Memos

Memos missing Deadlines

Strategy Risk & Control 78% 16 0

To Date £ Target Stretch Key Roles No. Filled Contractor %

Cost Reduction Resourcing 23 21 13%Cost Avoidance

Transformation Plans 0

Governance in place

TS Update Stakeholders mapped

Governance & Comms

OPERATIONAL

No. Lines Performance P&L YtoD Availability CSS

Service Lines ? AVAILABILITYE2E FIX SCOREGITS OVERALL CSS

Budget Forecast To date Variance Forecast To date Variance

Financial Ops. £2,222,333.44 £1,111,111.22 £1,233,222

No. No. Quality Improvement

Current Budgets

ForecastedExpenditure

% Projects Complete

£ Benefits

Projects

Revenue

Deliver irresistible value through...

Expenses

Inadequacies in dashboard population and scope of measurement

difficulties obtaining timely & accurate data

areas of performance left untracked (in scope & time)

deficient in leading indicators (heavily lagged-oriented): lack of predictive capacity to take proactive interventions

““I’ve no idea what drives the numbers. I’m not sure if anyone has” (manager)

Case: FindingsCase: Findings

lack of clarity or common understanding regarding definition of certain measures, e.g.

Constitution of measures: e.g. managed volume

(i) “We count managed volume against our target only when services have been transferred to GITS, and the first invoice sent to the end-market”;

(ii) “Managed volume is just that: services which we (GITS) manage. It doesn’t matter if we haven’t billed the customer yet.”

Progress towards targets: e.g. cost savings

(i) “We claim that we have achieved a cost saving when we sign a contract with an outsource provider to provide the service at a cost lower next year than our current deal”

(ii) “Cost savings are claimed when we release next years’ price list to the end markets in May, with confirmation in early December.”

Case: FindingsCase: Findings

Relation of measurement to strategy: difference between Leadership & AS/TS dashboards

Application Services & Technical Services dashboards reported self-chosen operational targets beyond existing strategy

Case: FindingsCase: Findings

non-strategy derived measures were seen by managers as useful aspects for monitoring operational health

“There are quite a few measures which don’t directly relate to strategy or targets, but we think it is worthwhile to keep track of them. It helps to know the operational health of the business.” (Application Services manager)

political value or ‘signaling’ ““We benchmark the charge rates of our project managers against external

consultancy providers; we’re less expensive, and the difference is classed as Cost Avoidance. It helps us demonstrate our value to the business.” (Technical Services manager)

Lack of systemisation in data collection and measurement (cum dashboard) design

no top-down mandate or formal programme / framework guiding the implementation of these practices: need for process rationalisation and information systems infrastructure (Bourne et. al, 2003)

difficulty identifying leading indicators (Neely et al., 2000; Eckerson, 2006)

Case: DiscussionCase: Discussion

Need for re-orientation of fundamental aim/focus

from a tool for simply monitoring/reporting to one of learning what factors drive results (i.e. to be able to influence/control)

Re-thinking major PM tenets/principles

1. Notion of ‘balance’ in balanced measurement

financial vs. non-financial ‘lever’ reporting vs. predication/learning ‘lever’ (lagging

vs. leading indicators)

Case: DiscussionCase: Discussion

2. Existing strategy as the source for deriving measures: a case for de-coupling strategy from measurement?

cost-focussed strategies promote financial and discourage non-financial indicators

‘de-politicizing’ of measurement promote transformation of PM towards a learning

rather than simply a monitoring tool

Results of this exploratory study suggest a need for further research & theoretical development to extend & deepen understanding of the complex nature of PM

What does ‘balanced’ measurement imply Relationship between strategy and measurement

Questions?

Thank you

ConclusionConclusion

END OF PRESENTATIONEND OF PRESENTATION

ReferencesReferences1.1. Kaplan, R. and Norton, D. (1992). Kaplan, R. and Norton, D. (1992). “The balanced scorecard – measures that drive “The balanced scorecard – measures that drive

performance.” Harvard Business Review. January-February, 71-29.performance.” Harvard Business Review. January-February, 71-29.

2.2. McCunn, P. (1998) The Balanced Scorecard: the eleventh commandment. McCunn, P. (1998) The Balanced Scorecard: the eleventh commandment. Management Accounting 34-36.Management Accounting 34-36.

3.3. Neely, A., Richards, H., Mills, J., Platts, K. and Bourne, M. (1997) Designing Neely, A., Richards, H., Mills, J., Platts, K. and Bourne, M. (1997) Designing performance measures: a structured approach. International Journal of performance measures: a structured approach. International Journal of

4.4. Operations & Production Management 17, 1131-1152.Operations & Production Management 17, 1131-1152.

5.5. De Toni, A. and Tonchia, S. (2001) Performance measurement systems - models, De Toni, A. and Tonchia, S. (2001) Performance measurement systems - models, characteristics and measures. International Journal of Productions and Operations characteristics and measures. International Journal of Productions and Operations Management 1, 347-354.Management 1, 347-354.

6.6. Eckerson, W. (2006) Performance Dashboards: Measuring Monitoring and Managing Eckerson, W. (2006) Performance Dashboards: Measuring Monitoring and Managing Your Business, edn. New Jersey: John Wiley & Sons.Your Business, edn. New Jersey: John Wiley & Sons.

7.7. Few, S. (2005) Dashboard Design: Beyond Meters, Gauges, ad Traffic Lights. Few, S. (2005) Dashboard Design: Beyond Meters, Gauges, ad Traffic Lights. Business Intelligence Journal 10, 18-24.Business Intelligence Journal 10, 18-24.

8.8. Dixon, J.R., Nanni, A.J. and Vollmann, T.E. (1990) The new performance challenge: Dixon, J.R., Nanni, A.J. and Vollmann, T.E. (1990) The new performance challenge: Measuring operations for world-class competition, Homewood, IL: Dow Jones-Irwin.Measuring operations for world-class competition, Homewood, IL: Dow Jones-Irwin.

9.9. Stanwick, P. and Stanwick, S. (2005) IT Performance: How Do You Measure a Moving Stanwick, P. and Stanwick, S. (2005) IT Performance: How Do You Measure a Moving Target? The Journal of Corporate Accounting and Finance 13, 19-24.Target? The Journal of Corporate Accounting and Finance 13, 19-24.

10.10. Walsham, G. (1995). Interpretive case studies in IS research: nature and method. Walsham, G. (1995). Interpretive case studies in IS research: nature and method. European Journal of Information Systems, 4:2, 74-81.European Journal of Information Systems, 4:2, 74-81.

11.11. Glaser, B. and Strauss, A. (1967) The discovery of grounded theory, Aldine, Chicago.Glaser, B. and Strauss, A. (1967) The discovery of grounded theory, Aldine, Chicago.

12.12. Bourne, M., Franco, M. and Wilkes, J. (2003) Corporate Performance Management. Bourne, M., Franco, M. and Wilkes, J. (2003) Corporate Performance Management. Measuring Business Excellence 7, 15-21.Measuring Business Excellence 7, 15-21.